Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 608048 Details for
Bug 852956
3.2 - prepareForShutdown is not called when connection to libvirt is broken with event: libvirtError: internal error client socket is closed
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
/var/log/vdsm/vdsm.log
vdsm.log (text/plain), 2.48 MB, created by
EricLee
on 2012-08-30 05:30:27 UTC
(
hide
)
Description:
/var/log/vdsm/vdsm.log
Filename:
MIME Type:
Creator:
EricLee
Created:
2012-08-30 05:30:27 UTC
Size:
2.48 MB
patch
obsolete
>Thread-1554::DEBUG::2012-08-30 12:46:09,483::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1554::DEBUG::2012-08-30 12:46:09,484::task::588::TaskManager.Task::(_updateState) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::moving from state init -> state preparing >Thread-1554::INFO::2012-08-30 12:46:09,484::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1554::DEBUG::2012-08-30 12:46:09,485::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`44070f95-838c-4ec2-8c06-f616231cfaa3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1554::DEBUG::2012-08-30 12:46:09,485::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1554::DEBUG::2012-08-30 12:46:09,485::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1554::DEBUG::2012-08-30 12:46:09,485::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`44070f95-838c-4ec2-8c06-f616231cfaa3`::Granted request >Thread-1554::DEBUG::2012-08-30 12:46:09,486::task::817::TaskManager.Task::(resourceAcquired) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1554::DEBUG::2012-08-30 12:46:09,486::task::978::TaskManager.Task::(_decref) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::ref 1 aborting False >Thread-1554::INFO::2012-08-30 12:46:09,489::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1554::DEBUG::2012-08-30 12:46:09,489::task::1172::TaskManager.Task::(prepare) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1554::DEBUG::2012-08-30 12:46:09,489::task::588::TaskManager.Task::(_updateState) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::moving from state preparing -> state finished >Thread-1554::DEBUG::2012-08-30 12:46:09,490::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1554::DEBUG::2012-08-30 12:46:09,490::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1554::DEBUG::2012-08-30 12:46:09,490::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1554::DEBUG::2012-08-30 12:46:09,490::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1554::DEBUG::2012-08-30 12:46:09,491::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1554::DEBUG::2012-08-30 12:46:09,491::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1554::DEBUG::2012-08-30 12:46:09,491::task::978::TaskManager.Task::(_decref) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::ref 0 aborting False >Thread-1555::DEBUG::2012-08-30 12:46:13,899::task::588::TaskManager.Task::(_updateState) Task=`9b887239-83c1-4d50-8567-339e0e2b4027`::moving from state init -> state preparing >Thread-1555::INFO::2012-08-30 12:46:13,900::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1555::INFO::2012-08-30 12:46:13,900::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164699554443', 'lastCheck': 1346301969.2454381, 'code': 0, 'valid': True}} >Thread-1555::DEBUG::2012-08-30 12:46:13,900::task::1172::TaskManager.Task::(prepare) Task=`9b887239-83c1-4d50-8567-339e0e2b4027`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164699554443', 'lastCheck': 1346301969.2454381, 'code': 0, 'valid': True}} >Thread-1555::DEBUG::2012-08-30 12:46:13,900::task::588::TaskManager.Task::(_updateState) Task=`9b887239-83c1-4d50-8567-339e0e2b4027`::moving from state preparing -> state finished >Thread-1555::DEBUG::2012-08-30 12:46:13,901::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1555::DEBUG::2012-08-30 12:46:13,901::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1555::DEBUG::2012-08-30 12:46:13,901::task::978::TaskManager.Task::(_decref) Task=`9b887239-83c1-4d50-8567-339e0e2b4027`::ref 0 aborting False >Thread-1556::DEBUG::2012-08-30 12:46:17,099::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1556::DEBUG::2012-08-30 12:46:17,100::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1558::DEBUG::2012-08-30 12:46:22,473::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1558::DEBUG::2012-08-30 12:46:22,473::task::588::TaskManager.Task::(_updateState) Task=`19d98c51-c2d9-4c41-9b5a-baaa85689617`::moving from state init -> state preparing >Thread-1558::INFO::2012-08-30 12:46:22,474::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1558::INFO::2012-08-30 12:46:22,474::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1558::DEBUG::2012-08-30 12:46:22,475::task::1172::TaskManager.Task::(prepare) Task=`19d98c51-c2d9-4c41-9b5a-baaa85689617`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1558::DEBUG::2012-08-30 12:46:22,475::task::588::TaskManager.Task::(_updateState) Task=`19d98c51-c2d9-4c41-9b5a-baaa85689617`::moving from state preparing -> state finished >Thread-1558::DEBUG::2012-08-30 12:46:22,475::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1558::DEBUG::2012-08-30 12:46:22,475::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1558::DEBUG::2012-08-30 12:46:22,476::task::978::TaskManager.Task::(_decref) Task=`19d98c51-c2d9-4c41-9b5a-baaa85689617`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:46:24,825::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1559::DEBUG::2012-08-30 12:46:25,693::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1559::DEBUG::2012-08-30 12:46:25,694::task::588::TaskManager.Task::(_updateState) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::moving from state init -> state preparing >Thread-1559::INFO::2012-08-30 12:46:25,694::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1559::DEBUG::2012-08-30 12:46:25,694::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`22bbf4fe-3219-4df4-aebb-b7067f524023`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1559::DEBUG::2012-08-30 12:46:25,694::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1559::DEBUG::2012-08-30 12:46:25,695::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1559::DEBUG::2012-08-30 12:46:25,695::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`22bbf4fe-3219-4df4-aebb-b7067f524023`::Granted request >Thread-1559::DEBUG::2012-08-30 12:46:25,695::task::817::TaskManager.Task::(resourceAcquired) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1559::DEBUG::2012-08-30 12:46:25,696::task::978::TaskManager.Task::(_decref) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::ref 1 aborting False >Thread-1559::INFO::2012-08-30 12:46:25,698::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1559::DEBUG::2012-08-30 12:46:25,699::task::1172::TaskManager.Task::(prepare) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1559::DEBUG::2012-08-30 12:46:25,699::task::588::TaskManager.Task::(_updateState) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::moving from state preparing -> state finished >Thread-1559::DEBUG::2012-08-30 12:46:25,699::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1559::DEBUG::2012-08-30 12:46:25,699::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1559::DEBUG::2012-08-30 12:46:25,700::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1559::DEBUG::2012-08-30 12:46:25,700::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1559::DEBUG::2012-08-30 12:46:25,700::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1559::DEBUG::2012-08-30 12:46:25,700::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1559::DEBUG::2012-08-30 12:46:25,701::task::978::TaskManager.Task::(_decref) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 12:46:29,971::task::588::TaskManager.Task::(_updateState) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:46:29,971::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:46:29,972::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`979971e2-984c-46ec-8a2f-814e2a814966`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:46:29,972::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:46:29,972::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:46:29,973::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`979971e2-984c-46ec-8a2f-814e2a814966`::Granted request >Thread-104::DEBUG::2012-08-30 12:46:29,973::task::817::TaskManager.Task::(resourceAcquired) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:46:29,973::task::978::TaskManager.Task::(_decref) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:46:29,975::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:46:29,977::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:46:29,978::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:46:29,978::task::1172::TaskManager.Task::(prepare) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:46:29,978::task::588::TaskManager.Task::(_updateState) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:46:29,979::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:46:29,979::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:46:29,979::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:46:29,979::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:46:29,980::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:46:29,980::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:46:29,980::task::978::TaskManager.Task::(_decref) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:46:31,832::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-104::ERROR::2012-08-30 12:46:34,994::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:46:37,953::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:46:37,954::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1563::DEBUG::2012-08-30 12:46:38,909::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1563::DEBUG::2012-08-30 12:46:38,910::task::588::TaskManager.Task::(_updateState) Task=`065c9c0c-1d8c-4558-8b13-44b3b53ffe7b`::moving from state init -> state preparing >Thread-1563::INFO::2012-08-30 12:46:38,910::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1563::INFO::2012-08-30 12:46:38,910::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1563::DEBUG::2012-08-30 12:46:38,911::task::1172::TaskManager.Task::(prepare) Task=`065c9c0c-1d8c-4558-8b13-44b3b53ffe7b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1563::DEBUG::2012-08-30 12:46:38,911::task::588::TaskManager.Task::(_updateState) Task=`065c9c0c-1d8c-4558-8b13-44b3b53ffe7b`::moving from state preparing -> state finished >Thread-1563::DEBUG::2012-08-30 12:46:38,911::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1563::DEBUG::2012-08-30 12:46:38,911::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1563::DEBUG::2012-08-30 12:46:38,911::task::978::TaskManager.Task::(_decref) Task=`065c9c0c-1d8c-4558-8b13-44b3b53ffe7b`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:46:40,001::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1564::DEBUG::2012-08-30 12:46:42,093::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1565::DEBUG::2012-08-30 12:46:42,093::task::588::TaskManager.Task::(_updateState) Task=`7fd8834c-7b37-4d76-b049-362778b088f1`::moving from state init -> state preparing >Thread-1564::DEBUG::2012-08-30 12:46:42,094::task::588::TaskManager.Task::(_updateState) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::moving from state init -> state preparing >Thread-1565::INFO::2012-08-30 12:46:42,094::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1564::INFO::2012-08-30 12:46:42,094::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1565::INFO::2012-08-30 12:46:42,094::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00248503684998', 'lastCheck': 1346301999.2711041, 'code': 0, 'valid': True}} >Thread-1564::DEBUG::2012-08-30 12:46:42,095::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9bab8c2a-220b-45d7-baea-96407cb7c7b0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1565::DEBUG::2012-08-30 12:46:42,095::task::1172::TaskManager.Task::(prepare) Task=`7fd8834c-7b37-4d76-b049-362778b088f1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00248503684998', 'lastCheck': 1346301999.2711041, 'code': 0, 'valid': True}} >Thread-1564::DEBUG::2012-08-30 12:46:42,095::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1565::DEBUG::2012-08-30 12:46:42,095::task::588::TaskManager.Task::(_updateState) Task=`7fd8834c-7b37-4d76-b049-362778b088f1`::moving from state preparing -> state finished >Thread-1565::DEBUG::2012-08-30 12:46:42,096::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1565::DEBUG::2012-08-30 12:46:42,097::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1564::DEBUG::2012-08-30 12:46:42,096::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1565::DEBUG::2012-08-30 12:46:42,097::task::978::TaskManager.Task::(_decref) Task=`7fd8834c-7b37-4d76-b049-362778b088f1`::ref 0 aborting False >Thread-1564::DEBUG::2012-08-30 12:46:42,097::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9bab8c2a-220b-45d7-baea-96407cb7c7b0`::Granted request >Thread-1564::DEBUG::2012-08-30 12:46:42,098::task::817::TaskManager.Task::(resourceAcquired) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1564::DEBUG::2012-08-30 12:46:42,098::task::978::TaskManager.Task::(_decref) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::ref 1 aborting False >Thread-1564::INFO::2012-08-30 12:46:42,103::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1564::DEBUG::2012-08-30 12:46:42,103::task::1172::TaskManager.Task::(prepare) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1564::DEBUG::2012-08-30 12:46:42,103::task::588::TaskManager.Task::(_updateState) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::moving from state preparing -> state finished >Thread-1564::DEBUG::2012-08-30 12:46:42,103::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1564::DEBUG::2012-08-30 12:46:42,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1564::DEBUG::2012-08-30 12:46:42,104::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1564::DEBUG::2012-08-30 12:46:42,104::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1564::DEBUG::2012-08-30 12:46:42,104::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1564::DEBUG::2012-08-30 12:46:42,105::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1564::DEBUG::2012-08-30 12:46:42,105::task::978::TaskManager.Task::(_decref) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:46:42,955::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:46:45,003::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:46:45,003::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1566::DEBUG::2012-08-30 12:46:45,285::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1566::DEBUG::2012-08-30 12:46:45,285::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 12:46:47,956::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:46:50,004::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 12:46:52,957::task::588::TaskManager.Task::(_updateState) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:46:52,957::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:46:52,958::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ad068212-b93b-4243-a101-734b4e83c5f3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:46:52,958::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:46:52,958::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:46:52,958::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ad068212-b93b-4243-a101-734b4e83c5f3`::Granted request >Thread-40::DEBUG::2012-08-30 12:46:52,959::task::817::TaskManager.Task::(resourceAcquired) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:46:52,959::task::978::TaskManager.Task::(_decref) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:46:52,961::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:46:52,963::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:46:52,964::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:46:52,964::task::1172::TaskManager.Task::(prepare) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:46:52,964::task::588::TaskManager.Task::(_updateState) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:46:52,965::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:46:52,965::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:46:52,965::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:46:52,965::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:46:52,966::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:46:52,966::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:46:52,966::task::978::TaskManager.Task::(_decref) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:46:52,966::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:46:52,967::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:46:52,967::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:46:52,968::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:46:54,851::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:46:55,006::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1568::DEBUG::2012-08-30 12:46:57,318::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1568::DEBUG::2012-08-30 12:46:57,319::task::588::TaskManager.Task::(_updateState) Task=`305bf8b3-a80f-4ce6-aaae-1788d2b00479`::moving from state init -> state preparing >Thread-1568::INFO::2012-08-30 12:46:57,319::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1568::INFO::2012-08-30 12:46:57,320::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1568::DEBUG::2012-08-30 12:46:57,320::task::1172::TaskManager.Task::(prepare) Task=`305bf8b3-a80f-4ce6-aaae-1788d2b00479`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1568::DEBUG::2012-08-30 12:46:57,320::task::588::TaskManager.Task::(_updateState) Task=`305bf8b3-a80f-4ce6-aaae-1788d2b00479`::moving from state preparing -> state finished >Thread-1568::DEBUG::2012-08-30 12:46:57,320::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1568::DEBUG::2012-08-30 12:46:57,321::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1568::DEBUG::2012-08-30 12:46:57,321::task::978::TaskManager.Task::(_decref) Task=`305bf8b3-a80f-4ce6-aaae-1788d2b00479`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:46:57,969::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:00,007::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:00,008::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1570::DEBUG::2012-08-30 12:47:00,511::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1570::DEBUG::2012-08-30 12:47:00,511::task::588::TaskManager.Task::(_updateState) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::moving from state init -> state preparing >Thread-1570::INFO::2012-08-30 12:47:00,512::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1570::DEBUG::2012-08-30 12:47:00,512::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e5189cb4-6c0f-4b7c-9937-61ae66d7e448`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1570::DEBUG::2012-08-30 12:47:00,512::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1570::DEBUG::2012-08-30 12:47:00,513::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1570::DEBUG::2012-08-30 12:47:00,513::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e5189cb4-6c0f-4b7c-9937-61ae66d7e448`::Granted request >Thread-1570::DEBUG::2012-08-30 12:47:00,513::task::817::TaskManager.Task::(resourceAcquired) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1570::DEBUG::2012-08-30 12:47:00,513::task::978::TaskManager.Task::(_decref) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::ref 1 aborting False >Thread-1570::INFO::2012-08-30 12:47:00,516::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1570::DEBUG::2012-08-30 12:47:00,516::task::1172::TaskManager.Task::(prepare) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1570::DEBUG::2012-08-30 12:47:00,517::task::588::TaskManager.Task::(_updateState) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::moving from state preparing -> state finished >Thread-1570::DEBUG::2012-08-30 12:47:00,517::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1570::DEBUG::2012-08-30 12:47:00,517::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1570::DEBUG::2012-08-30 12:47:00,517::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1570::DEBUG::2012-08-30 12:47:00,518::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1570::DEBUG::2012-08-30 12:47:00,518::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1570::DEBUG::2012-08-30 12:47:00,518::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1570::DEBUG::2012-08-30 12:47:00,518::task::978::TaskManager.Task::(_decref) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:47:01,858::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:47:02,970::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:05,009::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:07,971::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:07,971::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:10,010::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1573::DEBUG::2012-08-30 12:47:12,157::task::588::TaskManager.Task::(_updateState) Task=`2847ae63-91e1-4438-b6be-141f0c9f25a3`::moving from state init -> state preparing >Thread-1573::INFO::2012-08-30 12:47:12,157::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1573::INFO::2012-08-30 12:47:12,157::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162315368652', 'lastCheck': 1346302029.291816, 'code': 0, 'valid': True}} >Thread-1573::DEBUG::2012-08-30 12:47:12,158::task::1172::TaskManager.Task::(prepare) Task=`2847ae63-91e1-4438-b6be-141f0c9f25a3`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162315368652', 'lastCheck': 1346302029.291816, 'code': 0, 'valid': True}} >Thread-1573::DEBUG::2012-08-30 12:47:12,158::task::588::TaskManager.Task::(_updateState) Task=`2847ae63-91e1-4438-b6be-141f0c9f25a3`::moving from state preparing -> state finished >Thread-1573::DEBUG::2012-08-30 12:47:12,158::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1573::DEBUG::2012-08-30 12:47:12,158::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1573::DEBUG::2012-08-30 12:47:12,159::task::978::TaskManager.Task::(_decref) Task=`2847ae63-91e1-4438-b6be-141f0c9f25a3`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:47:12,972::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1574::DEBUG::2012-08-30 12:47:13,737::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1574::DEBUG::2012-08-30 12:47:13,738::task::588::TaskManager.Task::(_updateState) Task=`b0e70bef-48dc-401d-950a-d3878e592ccb`::moving from state init -> state preparing >Thread-1574::INFO::2012-08-30 12:47:13,738::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1574::INFO::2012-08-30 12:47:13,738::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1574::DEBUG::2012-08-30 12:47:13,738::task::1172::TaskManager.Task::(prepare) Task=`b0e70bef-48dc-401d-950a-d3878e592ccb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1574::DEBUG::2012-08-30 12:47:13,739::task::588::TaskManager.Task::(_updateState) Task=`b0e70bef-48dc-401d-950a-d3878e592ccb`::moving from state preparing -> state finished >Thread-1574::DEBUG::2012-08-30 12:47:13,739::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1574::DEBUG::2012-08-30 12:47:13,739::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1574::DEBUG::2012-08-30 12:47:13,739::task::978::TaskManager.Task::(_decref) Task=`b0e70bef-48dc-401d-950a-d3878e592ccb`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:47:15,012::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:15,012::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1575::DEBUG::2012-08-30 12:47:15,323::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1575::DEBUG::2012-08-30 12:47:15,323::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1576::DEBUG::2012-08-30 12:47:16,920::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1576::DEBUG::2012-08-30 12:47:16,920::task::588::TaskManager.Task::(_updateState) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::moving from state init -> state preparing >Thread-1576::INFO::2012-08-30 12:47:16,920::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1576::DEBUG::2012-08-30 12:47:16,921::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ee28c1d3-325e-4180-a0b2-0fb3b7c7e6ee`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1576::DEBUG::2012-08-30 12:47:16,921::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1576::DEBUG::2012-08-30 12:47:16,921::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1576::DEBUG::2012-08-30 12:47:16,921::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ee28c1d3-325e-4180-a0b2-0fb3b7c7e6ee`::Granted request >Thread-1576::DEBUG::2012-08-30 12:47:16,922::task::817::TaskManager.Task::(resourceAcquired) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1576::DEBUG::2012-08-30 12:47:16,922::task::978::TaskManager.Task::(_decref) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::ref 1 aborting False >Thread-1576::INFO::2012-08-30 12:47:16,925::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1576::DEBUG::2012-08-30 12:47:16,925::task::1172::TaskManager.Task::(prepare) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1576::DEBUG::2012-08-30 12:47:16,925::task::588::TaskManager.Task::(_updateState) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::moving from state preparing -> state finished >Thread-1576::DEBUG::2012-08-30 12:47:16,926::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1576::DEBUG::2012-08-30 12:47:16,926::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1576::DEBUG::2012-08-30 12:47:16,926::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1576::DEBUG::2012-08-30 12:47:16,926::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1576::DEBUG::2012-08-30 12:47:16,927::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1576::DEBUG::2012-08-30 12:47:16,927::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1576::DEBUG::2012-08-30 12:47:16,927::task::978::TaskManager.Task::(_decref) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:47:17,973::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:20,013::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:22,974::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:22,974::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:47:24,875::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:47:25,015::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:27,975::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1579::DEBUG::2012-08-30 12:47:29,497::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1579::DEBUG::2012-08-30 12:47:29,498::task::588::TaskManager.Task::(_updateState) Task=`75dc896c-93c3-421b-8fc5-e88a7d08874f`::moving from state init -> state preparing >Thread-1579::INFO::2012-08-30 12:47:29,498::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1579::INFO::2012-08-30 12:47:29,498::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1579::DEBUG::2012-08-30 12:47:29,499::task::1172::TaskManager.Task::(prepare) Task=`75dc896c-93c3-421b-8fc5-e88a7d08874f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1579::DEBUG::2012-08-30 12:47:29,499::task::588::TaskManager.Task::(_updateState) Task=`75dc896c-93c3-421b-8fc5-e88a7d08874f`::moving from state preparing -> state finished >Thread-1579::DEBUG::2012-08-30 12:47:29,499::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1579::DEBUG::2012-08-30 12:47:29,499::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1579::DEBUG::2012-08-30 12:47:29,500::task::978::TaskManager.Task::(_decref) Task=`75dc896c-93c3-421b-8fc5-e88a7d08874f`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 12:47:30,016::task::588::TaskManager.Task::(_updateState) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:47:30,016::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:47:30,017::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a5c9b71c-250d-479c-af49-13dab4e8b200`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:47:30,017::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:47:30,017::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:47:30,017::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a5c9b71c-250d-479c-af49-13dab4e8b200`::Granted request >Thread-104::DEBUG::2012-08-30 12:47:30,018::task::817::TaskManager.Task::(resourceAcquired) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:47:30,018::task::978::TaskManager.Task::(_decref) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:47:30,020::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:47:30,022::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:47:30,023::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:47:30,023::task::1172::TaskManager.Task::(prepare) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:47:30,023::task::588::TaskManager.Task::(_updateState) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:47:30,023::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:47:30,024::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:47:30,024::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:47:30,024::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:47:30,024::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:47:30,025::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:47:30,025::task::978::TaskManager.Task::(_decref) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:47:30,025::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:30,026::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:30,026::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:30,027::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:47:31,882::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1581::DEBUG::2012-08-30 12:47:32,703::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1581::DEBUG::2012-08-30 12:47:32,704::task::588::TaskManager.Task::(_updateState) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::moving from state init -> state preparing >Thread-1581::INFO::2012-08-30 12:47:32,704::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1581::DEBUG::2012-08-30 12:47:32,704::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6ea65cbb-aff7-432f-af13-22f56d8f0355`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1581::DEBUG::2012-08-30 12:47:32,705::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1581::DEBUG::2012-08-30 12:47:32,705::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1581::DEBUG::2012-08-30 12:47:32,705::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6ea65cbb-aff7-432f-af13-22f56d8f0355`::Granted request >Thread-1581::DEBUG::2012-08-30 12:47:32,705::task::817::TaskManager.Task::(resourceAcquired) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1581::DEBUG::2012-08-30 12:47:32,706::task::978::TaskManager.Task::(_decref) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::ref 1 aborting False >Thread-1581::INFO::2012-08-30 12:47:32,709::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1581::DEBUG::2012-08-30 12:47:32,709::task::1172::TaskManager.Task::(prepare) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1581::DEBUG::2012-08-30 12:47:32,709::task::588::TaskManager.Task::(_updateState) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::moving from state preparing -> state finished >Thread-1581::DEBUG::2012-08-30 12:47:32,709::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1581::DEBUG::2012-08-30 12:47:32,711::task::978::TaskManager.Task::(_decref) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:47:32,976::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:35,028::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:37,977::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:37,978::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:40,029::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1583::DEBUG::2012-08-30 12:47:41,501::task::588::TaskManager.Task::(_updateState) Task=`ace23e47-08a9-4b25-8a57-a9e00c014c28`::moving from state init -> state preparing >Thread-1583::INFO::2012-08-30 12:47:41,501::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1583::INFO::2012-08-30 12:47:41,501::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161981582642', 'lastCheck': 1346302059.3110631, 'code': 0, 'valid': True}} >Thread-1583::DEBUG::2012-08-30 12:47:41,502::task::1172::TaskManager.Task::(prepare) Task=`ace23e47-08a9-4b25-8a57-a9e00c014c28`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161981582642', 'lastCheck': 1346302059.3110631, 'code': 0, 'valid': True}} >Thread-1583::DEBUG::2012-08-30 12:47:41,502::task::588::TaskManager.Task::(_updateState) Task=`ace23e47-08a9-4b25-8a57-a9e00c014c28`::moving from state preparing -> state finished >Thread-1583::DEBUG::2012-08-30 12:47:41,502::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1583::DEBUG::2012-08-30 12:47:41,502::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1583::DEBUG::2012-08-30 12:47:41,503::task::978::TaskManager.Task::(_decref) Task=`ace23e47-08a9-4b25-8a57-a9e00c014c28`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:47:42,979::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1584::DEBUG::2012-08-30 12:47:44,351::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1584::DEBUG::2012-08-30 12:47:44,352::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:47:45,031::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:47:45,031::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1585::DEBUG::2012-08-30 12:47:45,941::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1585::DEBUG::2012-08-30 12:47:45,941::task::588::TaskManager.Task::(_updateState) Task=`a565f7e6-7803-4e6b-97c2-1993312cd86f`::moving from state init -> state preparing >Thread-1585::INFO::2012-08-30 12:47:45,941::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1585::INFO::2012-08-30 12:47:45,942::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1585::DEBUG::2012-08-30 12:47:45,942::task::1172::TaskManager.Task::(prepare) Task=`a565f7e6-7803-4e6b-97c2-1993312cd86f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1585::DEBUG::2012-08-30 12:47:45,942::task::588::TaskManager.Task::(_updateState) Task=`a565f7e6-7803-4e6b-97c2-1993312cd86f`::moving from state preparing -> state finished >Thread-1585::DEBUG::2012-08-30 12:47:45,942::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1585::DEBUG::2012-08-30 12:47:45,943::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1585::DEBUG::2012-08-30 12:47:45,943::task::978::TaskManager.Task::(_decref) Task=`a565f7e6-7803-4e6b-97c2-1993312cd86f`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:47:47,980::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1586::DEBUG::2012-08-30 12:47:49,115::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1586::DEBUG::2012-08-30 12:47:49,116::task::588::TaskManager.Task::(_updateState) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::moving from state init -> state preparing >Thread-1586::INFO::2012-08-30 12:47:49,116::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1586::DEBUG::2012-08-30 12:47:49,117::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`73053b8f-aade-4b88-b589-2f41888056fb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1586::DEBUG::2012-08-30 12:47:49,117::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1586::DEBUG::2012-08-30 12:47:49,117::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1586::DEBUG::2012-08-30 12:47:49,117::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`73053b8f-aade-4b88-b589-2f41888056fb`::Granted request >Thread-1586::DEBUG::2012-08-30 12:47:49,118::task::817::TaskManager.Task::(resourceAcquired) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1586::DEBUG::2012-08-30 12:47:49,118::task::978::TaskManager.Task::(_decref) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::ref 1 aborting False >Thread-1586::INFO::2012-08-30 12:47:49,121::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1586::DEBUG::2012-08-30 12:47:49,121::task::1172::TaskManager.Task::(prepare) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1586::DEBUG::2012-08-30 12:47:49,121::task::588::TaskManager.Task::(_updateState) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::moving from state preparing -> state finished >Thread-1586::DEBUG::2012-08-30 12:47:49,122::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1586::DEBUG::2012-08-30 12:47:49,122::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1586::DEBUG::2012-08-30 12:47:49,122::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1586::DEBUG::2012-08-30 12:47:49,122::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1586::DEBUG::2012-08-30 12:47:49,123::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1586::DEBUG::2012-08-30 12:47:49,123::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1586::DEBUG::2012-08-30 12:47:49,123::task::978::TaskManager.Task::(_decref) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:47:50,033::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 12:47:52,981::task::588::TaskManager.Task::(_updateState) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:47:52,981::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:47:52,981::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`11e3767a-4fb5-4612-b113-a5a412899247`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:47:52,981::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:47:52,982::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:47:52,982::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`11e3767a-4fb5-4612-b113-a5a412899247`::Granted request >Thread-40::DEBUG::2012-08-30 12:47:52,982::task::817::TaskManager.Task::(resourceAcquired) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:47:52,983::task::978::TaskManager.Task::(_decref) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:47:52,984::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:47:52,987::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:47:52,988::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:47:52,988::task::1172::TaskManager.Task::(prepare) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:47:52,988::task::588::TaskManager.Task::(_updateState) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:47:52,988::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:47:52,990::task::978::TaskManager.Task::(_decref) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:47:52,990::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:52,991::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:52,991::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:52,991::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:47:54,905::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:47:55,034::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:47:57,992::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:00,035::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:00,036::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:48:01,911::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1590::DEBUG::2012-08-30 12:48:02,360::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1590::DEBUG::2012-08-30 12:48:02,360::task::588::TaskManager.Task::(_updateState) Task=`8774e142-89b6-42e4-81fa-1c3eb05e8a8c`::moving from state init -> state preparing >Thread-1590::INFO::2012-08-30 12:48:02,360::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1590::INFO::2012-08-30 12:48:02,361::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1590::DEBUG::2012-08-30 12:48:02,361::task::1172::TaskManager.Task::(prepare) Task=`8774e142-89b6-42e4-81fa-1c3eb05e8a8c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1590::DEBUG::2012-08-30 12:48:02,361::task::588::TaskManager.Task::(_updateState) Task=`8774e142-89b6-42e4-81fa-1c3eb05e8a8c`::moving from state preparing -> state finished >Thread-1590::DEBUG::2012-08-30 12:48:02,361::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1590::DEBUG::2012-08-30 12:48:02,362::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1590::DEBUG::2012-08-30 12:48:02,362::task::978::TaskManager.Task::(_decref) Task=`8774e142-89b6-42e4-81fa-1c3eb05e8a8c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:48:02,993::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:05,037::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1592::DEBUG::2012-08-30 12:48:05,564::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1592::DEBUG::2012-08-30 12:48:05,565::task::588::TaskManager.Task::(_updateState) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::moving from state init -> state preparing >Thread-1592::INFO::2012-08-30 12:48:05,565::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1592::DEBUG::2012-08-30 12:48:05,565::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`cafb36f9-b0c9-448b-80c4-614fe06163ba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1592::DEBUG::2012-08-30 12:48:05,566::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1592::DEBUG::2012-08-30 12:48:05,566::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1592::DEBUG::2012-08-30 12:48:05,566::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`cafb36f9-b0c9-448b-80c4-614fe06163ba`::Granted request >Thread-1592::DEBUG::2012-08-30 12:48:05,566::task::817::TaskManager.Task::(resourceAcquired) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1592::DEBUG::2012-08-30 12:48:05,567::task::978::TaskManager.Task::(_decref) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::ref 1 aborting False >Thread-1592::INFO::2012-08-30 12:48:05,570::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1592::DEBUG::2012-08-30 12:48:05,570::task::1172::TaskManager.Task::(prepare) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1592::DEBUG::2012-08-30 12:48:05,570::task::588::TaskManager.Task::(_updateState) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::moving from state preparing -> state finished >Thread-1592::DEBUG::2012-08-30 12:48:05,570::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1592::DEBUG::2012-08-30 12:48:05,572::task::978::TaskManager.Task::(_decref) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:48:07,994::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:07,995::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1593::DEBUG::2012-08-30 12:48:09,939::task::588::TaskManager.Task::(_updateState) Task=`053dd472-8fc9-4b69-a5cb-489330f04201`::moving from state init -> state preparing >Thread-1593::INFO::2012-08-30 12:48:09,939::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1593::INFO::2012-08-30 12:48:09,939::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346302089.3315029, 'code': 0, 'valid': True}} >Thread-1593::DEBUG::2012-08-30 12:48:09,940::task::1172::TaskManager.Task::(prepare) Task=`053dd472-8fc9-4b69-a5cb-489330f04201`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346302089.3315029, 'code': 0, 'valid': True}} >Thread-1593::DEBUG::2012-08-30 12:48:09,940::task::588::TaskManager.Task::(_updateState) Task=`053dd472-8fc9-4b69-a5cb-489330f04201`::moving from state preparing -> state finished >Thread-1593::DEBUG::2012-08-30 12:48:09,940::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1593::DEBUG::2012-08-30 12:48:09,940::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1593::DEBUG::2012-08-30 12:48:09,940::task::978::TaskManager.Task::(_decref) Task=`053dd472-8fc9-4b69-a5cb-489330f04201`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:48:10,038::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:12,995::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1594::DEBUG::2012-08-30 12:48:13,347::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1594::DEBUG::2012-08-30 12:48:13,347::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:48:15,039::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:15,040::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:17,996::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1596::DEBUG::2012-08-30 12:48:18,623::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1596::DEBUG::2012-08-30 12:48:18,624::task::588::TaskManager.Task::(_updateState) Task=`f858818b-d1e8-4c95-a47f-36dbc40f7286`::moving from state init -> state preparing >Thread-1596::INFO::2012-08-30 12:48:18,624::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1596::INFO::2012-08-30 12:48:18,625::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1596::DEBUG::2012-08-30 12:48:18,625::task::1172::TaskManager.Task::(prepare) Task=`f858818b-d1e8-4c95-a47f-36dbc40f7286`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1596::DEBUG::2012-08-30 12:48:18,625::task::588::TaskManager.Task::(_updateState) Task=`f858818b-d1e8-4c95-a47f-36dbc40f7286`::moving from state preparing -> state finished >Thread-1596::DEBUG::2012-08-30 12:48:18,625::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1596::DEBUG::2012-08-30 12:48:18,626::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1596::DEBUG::2012-08-30 12:48:18,626::task::978::TaskManager.Task::(_decref) Task=`f858818b-d1e8-4c95-a47f-36dbc40f7286`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:48:20,041::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1597::DEBUG::2012-08-30 12:48:21,828::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1597::DEBUG::2012-08-30 12:48:21,829::task::588::TaskManager.Task::(_updateState) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::moving from state init -> state preparing >Thread-1597::INFO::2012-08-30 12:48:21,829::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1597::DEBUG::2012-08-30 12:48:21,830::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8c87deda-5c0f-4208-bb61-1d66f9a9ccc1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1597::DEBUG::2012-08-30 12:48:21,830::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1597::DEBUG::2012-08-30 12:48:21,830::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1597::DEBUG::2012-08-30 12:48:21,830::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8c87deda-5c0f-4208-bb61-1d66f9a9ccc1`::Granted request >Thread-1597::DEBUG::2012-08-30 12:48:21,831::task::817::TaskManager.Task::(resourceAcquired) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1597::DEBUG::2012-08-30 12:48:21,831::task::978::TaskManager.Task::(_decref) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::ref 1 aborting False >Thread-1597::INFO::2012-08-30 12:48:21,834::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1597::DEBUG::2012-08-30 12:48:21,834::task::1172::TaskManager.Task::(prepare) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1597::DEBUG::2012-08-30 12:48:21,834::task::588::TaskManager.Task::(_updateState) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::moving from state preparing -> state finished >Thread-1597::DEBUG::2012-08-30 12:48:21,835::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1597::DEBUG::2012-08-30 12:48:21,835::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1597::DEBUG::2012-08-30 12:48:21,835::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1597::DEBUG::2012-08-30 12:48:21,835::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1597::DEBUG::2012-08-30 12:48:21,836::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1597::DEBUG::2012-08-30 12:48:21,836::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1597::DEBUG::2012-08-30 12:48:21,836::task::978::TaskManager.Task::(_decref) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:48:22,997::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:22,998::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:48:24,933::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:48:25,042::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:27,999::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:48:30,043::task::588::TaskManager.Task::(_updateState) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:48:30,044::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:48:30,044::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`476a2dae-0fc4-4390-ba9b-7f0309c2b10b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:48:30,044::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:48:30,045::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:48:30,045::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`476a2dae-0fc4-4390-ba9b-7f0309c2b10b`::Granted request >Thread-104::DEBUG::2012-08-30 12:48:30,045::task::817::TaskManager.Task::(resourceAcquired) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:48:30,045::task::978::TaskManager.Task::(_decref) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:48:30,047::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:48:30,049::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:48:30,050::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:48:30,050::task::1172::TaskManager.Task::(prepare) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:48:30,051::task::588::TaskManager.Task::(_updateState) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:48:30,051::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:48:30,051::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:48:30,051::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:48:30,052::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:48:30,052::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:48:30,052::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:48:30,052::task::978::TaskManager.Task::(_decref) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:48:30,053::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:30,053::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:30,053::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:30,054::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:48:31,940::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:48:33,000::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:35,055::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1601::DEBUG::2012-08-30 12:48:35,082::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1601::DEBUG::2012-08-30 12:48:35,083::task::588::TaskManager.Task::(_updateState) Task=`e7d6243e-039a-4ea8-abea-69594a335359`::moving from state init -> state preparing >Thread-1601::INFO::2012-08-30 12:48:35,083::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1601::INFO::2012-08-30 12:48:35,083::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1601::DEBUG::2012-08-30 12:48:35,084::task::1172::TaskManager.Task::(prepare) Task=`e7d6243e-039a-4ea8-abea-69594a335359`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1601::DEBUG::2012-08-30 12:48:35,084::task::588::TaskManager.Task::(_updateState) Task=`e7d6243e-039a-4ea8-abea-69594a335359`::moving from state preparing -> state finished >Thread-1601::DEBUG::2012-08-30 12:48:35,084::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1601::DEBUG::2012-08-30 12:48:35,084::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1601::DEBUG::2012-08-30 12:48:35,085::task::978::TaskManager.Task::(_decref) Task=`e7d6243e-039a-4ea8-abea-69594a335359`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:48:38,000::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:38,001::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1602::DEBUG::2012-08-30 12:48:38,265::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1603::DEBUG::2012-08-30 12:48:38,265::task::588::TaskManager.Task::(_updateState) Task=`96fe9764-0b7f-409a-9ae7-f763e05c1be8`::moving from state init -> state preparing >Thread-1602::DEBUG::2012-08-30 12:48:38,265::task::588::TaskManager.Task::(_updateState) Task=`a410bb23-2bce-4806-88e9-40875c336007`::moving from state init -> state preparing >Thread-1603::INFO::2012-08-30 12:48:38,266::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1602::INFO::2012-08-30 12:48:38,266::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1603::INFO::2012-08-30 12:48:38,266::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163698196411', 'lastCheck': 1346302109.3441539, 'code': 0, 'valid': True}} >Thread-1602::DEBUG::2012-08-30 12:48:38,266::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`7676593c-682b-48c3-9ff2-9122225f2b21`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1603::DEBUG::2012-08-30 12:48:38,267::task::1172::TaskManager.Task::(prepare) Task=`96fe9764-0b7f-409a-9ae7-f763e05c1be8`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163698196411', 'lastCheck': 1346302109.3441539, 'code': 0, 'valid': True}} >Thread-1602::DEBUG::2012-08-30 12:48:38,267::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1603::DEBUG::2012-08-30 12:48:38,267::task::588::TaskManager.Task::(_updateState) Task=`96fe9764-0b7f-409a-9ae7-f763e05c1be8`::moving from state preparing -> state finished >Thread-1602::DEBUG::2012-08-30 12:48:38,267::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1603::DEBUG::2012-08-30 12:48:38,267::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1602::DEBUG::2012-08-30 12:48:38,268::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`7676593c-682b-48c3-9ff2-9122225f2b21`::Granted request >Thread-1603::DEBUG::2012-08-30 12:48:38,268::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1602::DEBUG::2012-08-30 12:48:38,268::task::817::TaskManager.Task::(resourceAcquired) Task=`a410bb23-2bce-4806-88e9-40875c336007`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1603::DEBUG::2012-08-30 12:48:38,268::task::978::TaskManager.Task::(_decref) Task=`96fe9764-0b7f-409a-9ae7-f763e05c1be8`::ref 0 aborting False >Thread-1602::DEBUG::2012-08-30 12:48:38,269::task::978::TaskManager.Task::(_decref) Task=`a410bb23-2bce-4806-88e9-40875c336007`::ref 1 aborting False >Thread-1602::INFO::2012-08-30 12:48:38,273::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1602::DEBUG::2012-08-30 12:48:38,274::task::1172::TaskManager.Task::(prepare) Task=`a410bb23-2bce-4806-88e9-40875c336007`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1602::DEBUG::2012-08-30 12:48:38,274::task::588::TaskManager.Task::(_updateState) Task=`a410bb23-2bce-4806-88e9-40875c336007`::moving from state preparing -> state finished >Thread-1602::DEBUG::2012-08-30 12:48:38,274::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1602::DEBUG::2012-08-30 12:48:38,274::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1602::DEBUG::2012-08-30 12:48:38,275::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1602::DEBUG::2012-08-30 12:48:38,275::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1602::DEBUG::2012-08-30 12:48:38,275::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1602::DEBUG::2012-08-30 12:48:38,275::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1602::DEBUG::2012-08-30 12:48:38,276::task::978::TaskManager.Task::(_decref) Task=`a410bb23-2bce-4806-88e9-40875c336007`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:48:40,056::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1604::DEBUG::2012-08-30 12:48:41,473::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1604::DEBUG::2012-08-30 12:48:41,474::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 12:48:43,002::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:45,057::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:45,058::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:48,003::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:48:50,059::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1606::DEBUG::2012-08-30 12:48:51,643::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1606::DEBUG::2012-08-30 12:48:51,644::task::588::TaskManager.Task::(_updateState) Task=`d676892e-e340-4076-94b4-ad5321e9df5c`::moving from state init -> state preparing >Thread-1606::INFO::2012-08-30 12:48:51,644::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1606::INFO::2012-08-30 12:48:51,644::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1606::DEBUG::2012-08-30 12:48:51,645::task::1172::TaskManager.Task::(prepare) Task=`d676892e-e340-4076-94b4-ad5321e9df5c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1606::DEBUG::2012-08-30 12:48:51,645::task::588::TaskManager.Task::(_updateState) Task=`d676892e-e340-4076-94b4-ad5321e9df5c`::moving from state preparing -> state finished >Thread-1606::DEBUG::2012-08-30 12:48:51,645::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1606::DEBUG::2012-08-30 12:48:51,645::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1606::DEBUG::2012-08-30 12:48:51,646::task::978::TaskManager.Task::(_decref) Task=`d676892e-e340-4076-94b4-ad5321e9df5c`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 12:48:53,004::task::588::TaskManager.Task::(_updateState) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:48:53,004::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:48:53,004::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`599a8e2c-2172-4941-8252-75cdcfc0e440`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:48:53,005::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:48:53,005::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:48:53,005::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`599a8e2c-2172-4941-8252-75cdcfc0e440`::Granted request >Thread-40::DEBUG::2012-08-30 12:48:53,006::task::817::TaskManager.Task::(resourceAcquired) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:48:53,006::task::978::TaskManager.Task::(_decref) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:48:53,008::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:48:53,010::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:48:53,011::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:48:53,011::task::1172::TaskManager.Task::(prepare) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:48:53,011::task::588::TaskManager.Task::(_updateState) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:48:53,012::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:48:53,012::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:48:53,012::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:48:53,012::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:48:53,013::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:48:53,013::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:48:53,013::task::978::TaskManager.Task::(_decref) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:48:53,013::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:53,014::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:53,014::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:53,015::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1608::DEBUG::2012-08-30 12:48:54,815::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1608::DEBUG::2012-08-30 12:48:54,815::task::588::TaskManager.Task::(_updateState) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::moving from state init -> state preparing >Thread-1608::INFO::2012-08-30 12:48:54,816::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1608::DEBUG::2012-08-30 12:48:54,816::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8b29fa7b-a030-4f43-ad1d-de2b9c0e7da4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1608::DEBUG::2012-08-30 12:48:54,816::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1608::DEBUG::2012-08-30 12:48:54,817::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1608::DEBUG::2012-08-30 12:48:54,817::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8b29fa7b-a030-4f43-ad1d-de2b9c0e7da4`::Granted request >Thread-1608::DEBUG::2012-08-30 12:48:54,817::task::817::TaskManager.Task::(resourceAcquired) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1608::DEBUG::2012-08-30 12:48:54,817::task::978::TaskManager.Task::(_decref) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::ref 1 aborting False >Thread-1608::INFO::2012-08-30 12:48:54,820::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1608::DEBUG::2012-08-30 12:48:54,821::task::1172::TaskManager.Task::(prepare) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1608::DEBUG::2012-08-30 12:48:54,821::task::588::TaskManager.Task::(_updateState) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::moving from state preparing -> state finished >Thread-1608::DEBUG::2012-08-30 12:48:54,821::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1608::DEBUG::2012-08-30 12:48:54,821::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1608::DEBUG::2012-08-30 12:48:54,822::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1608::DEBUG::2012-08-30 12:48:54,822::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1608::DEBUG::2012-08-30 12:48:54,822::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1608::DEBUG::2012-08-30 12:48:54,822::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1608::DEBUG::2012-08-30 12:48:54,822::task::978::TaskManager.Task::(_decref) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:48:54,958::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:48:55,060::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:48:58,016::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:00,062::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:00,062::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:49:01,966::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:49:03,017::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:05,063::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1611::DEBUG::2012-08-30 12:49:06,463::task::588::TaskManager.Task::(_updateState) Task=`73c4895c-4ea6-449a-af4b-cbe4ff42fea7`::moving from state init -> state preparing >Thread-1611::INFO::2012-08-30 12:49:06,464::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1611::INFO::2012-08-30 12:49:06,464::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178503990173', 'lastCheck': 1346302139.36462, 'code': 0, 'valid': True}} >Thread-1611::DEBUG::2012-08-30 12:49:06,464::task::1172::TaskManager.Task::(prepare) Task=`73c4895c-4ea6-449a-af4b-cbe4ff42fea7`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178503990173', 'lastCheck': 1346302139.36462, 'code': 0, 'valid': True}} >Thread-1611::DEBUG::2012-08-30 12:49:06,464::task::588::TaskManager.Task::(_updateState) Task=`73c4895c-4ea6-449a-af4b-cbe4ff42fea7`::moving from state preparing -> state finished >Thread-1611::DEBUG::2012-08-30 12:49:06,464::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1611::DEBUG::2012-08-30 12:49:06,465::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1611::DEBUG::2012-08-30 12:49:06,465::task::978::TaskManager.Task::(_decref) Task=`73c4895c-4ea6-449a-af4b-cbe4ff42fea7`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:49:08,018::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:08,018::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1612::DEBUG::2012-08-30 12:49:08,054::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1612::DEBUG::2012-08-30 12:49:08,054::task::588::TaskManager.Task::(_updateState) Task=`ecccd816-c460-47d9-80b3-84f57e64b169`::moving from state init -> state preparing >Thread-1612::INFO::2012-08-30 12:49:08,055::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1612::INFO::2012-08-30 12:49:08,055::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1612::DEBUG::2012-08-30 12:49:08,055::task::1172::TaskManager.Task::(prepare) Task=`ecccd816-c460-47d9-80b3-84f57e64b169`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1612::DEBUG::2012-08-30 12:49:08,055::task::588::TaskManager.Task::(_updateState) Task=`ecccd816-c460-47d9-80b3-84f57e64b169`::moving from state preparing -> state finished >Thread-1612::DEBUG::2012-08-30 12:49:08,056::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1612::DEBUG::2012-08-30 12:49:08,056::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1612::DEBUG::2012-08-30 12:49:08,056::task::978::TaskManager.Task::(_decref) Task=`ecccd816-c460-47d9-80b3-84f57e64b169`::ref 0 aborting False >Thread-1613::DEBUG::2012-08-30 12:49:09,938::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1613::DEBUG::2012-08-30 12:49:09,938::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:49:10,064::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1614::DEBUG::2012-08-30 12:49:11,530::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1614::DEBUG::2012-08-30 12:49:11,531::task::588::TaskManager.Task::(_updateState) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::moving from state init -> state preparing >Thread-1614::INFO::2012-08-30 12:49:11,531::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1614::DEBUG::2012-08-30 12:49:11,531::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5fa49e40-e9a7-4b83-b3a5-91140095bce8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1614::DEBUG::2012-08-30 12:49:11,532::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1614::DEBUG::2012-08-30 12:49:11,532::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1614::DEBUG::2012-08-30 12:49:11,532::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5fa49e40-e9a7-4b83-b3a5-91140095bce8`::Granted request >Thread-1614::DEBUG::2012-08-30 12:49:11,532::task::817::TaskManager.Task::(resourceAcquired) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1614::DEBUG::2012-08-30 12:49:11,533::task::978::TaskManager.Task::(_decref) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::ref 1 aborting False >Thread-1614::INFO::2012-08-30 12:49:11,536::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1614::DEBUG::2012-08-30 12:49:11,536::task::1172::TaskManager.Task::(prepare) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1614::DEBUG::2012-08-30 12:49:11,536::task::588::TaskManager.Task::(_updateState) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::moving from state preparing -> state finished >Thread-1614::DEBUG::2012-08-30 12:49:11,536::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1614::DEBUG::2012-08-30 12:49:11,538::task::978::TaskManager.Task::(_decref) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:49:13,019::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:15,066::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:15,066::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:18,020::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:20,067::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:23,021::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:23,021::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1617::DEBUG::2012-08-30 12:49:24,734::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1617::DEBUG::2012-08-30 12:49:24,735::task::588::TaskManager.Task::(_updateState) Task=`77695686-65ff-4d7e-ac8c-9c839784809a`::moving from state init -> state preparing >Thread-1617::INFO::2012-08-30 12:49:24,735::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1617::INFO::2012-08-30 12:49:24,736::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1617::DEBUG::2012-08-30 12:49:24,736::task::1172::TaskManager.Task::(prepare) Task=`77695686-65ff-4d7e-ac8c-9c839784809a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1617::DEBUG::2012-08-30 12:49:24,736::task::588::TaskManager.Task::(_updateState) Task=`77695686-65ff-4d7e-ac8c-9c839784809a`::moving from state preparing -> state finished >Thread-1617::DEBUG::2012-08-30 12:49:24,736::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1617::DEBUG::2012-08-30 12:49:24,736::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1617::DEBUG::2012-08-30 12:49:24,737::task::978::TaskManager.Task::(_decref) Task=`77695686-65ff-4d7e-ac8c-9c839784809a`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:49:24,985::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:49:25,069::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1619::DEBUG::2012-08-30 12:49:27,916::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1619::DEBUG::2012-08-30 12:49:27,917::task::588::TaskManager.Task::(_updateState) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::moving from state init -> state preparing >Thread-1619::INFO::2012-08-30 12:49:27,917::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1619::DEBUG::2012-08-30 12:49:27,917::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`54593cdb-ddb3-4fc0-9e6f-aea678e3a1a4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1619::DEBUG::2012-08-30 12:49:27,918::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1619::DEBUG::2012-08-30 12:49:27,918::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1619::DEBUG::2012-08-30 12:49:27,918::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`54593cdb-ddb3-4fc0-9e6f-aea678e3a1a4`::Granted request >Thread-1619::DEBUG::2012-08-30 12:49:27,919::task::817::TaskManager.Task::(resourceAcquired) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1619::DEBUG::2012-08-30 12:49:27,919::task::978::TaskManager.Task::(_decref) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::ref 1 aborting False >Thread-1619::INFO::2012-08-30 12:49:27,922::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1619::DEBUG::2012-08-30 12:49:27,922::task::1172::TaskManager.Task::(prepare) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1619::DEBUG::2012-08-30 12:49:27,922::task::588::TaskManager.Task::(_updateState) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::moving from state preparing -> state finished >Thread-1619::DEBUG::2012-08-30 12:49:27,922::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1619::DEBUG::2012-08-30 12:49:27,923::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1619::DEBUG::2012-08-30 12:49:27,923::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1619::DEBUG::2012-08-30 12:49:27,923::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1619::DEBUG::2012-08-30 12:49:27,923::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1619::DEBUG::2012-08-30 12:49:27,924::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1619::DEBUG::2012-08-30 12:49:27,924::task::978::TaskManager.Task::(_decref) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:49:28,022::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:49:30,070::task::588::TaskManager.Task::(_updateState) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:49:30,070::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:49:30,070::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`550455b7-4232-4de6-8504-514c1ccd9f2a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:49:30,071::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:49:30,071::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:49:30,071::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`550455b7-4232-4de6-8504-514c1ccd9f2a`::Granted request >Thread-104::DEBUG::2012-08-30 12:49:30,071::task::817::TaskManager.Task::(resourceAcquired) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:49:30,072::task::978::TaskManager.Task::(_decref) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:49:30,073::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:49:30,075::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:49:30,076::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:49:30,077::task::1172::TaskManager.Task::(prepare) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:49:30,077::task::588::TaskManager.Task::(_updateState) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:49:30,077::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:49:30,077::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:49:30,078::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:49:30,078::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:49:30,078::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:49:30,078::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:49:30,078::task::978::TaskManager.Task::(_decref) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:49:30,079::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:30,079::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:30,080::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:30,080::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:49:31,990::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:49:33,023::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:35,081::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:38,024::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:38,024::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1621::DEBUG::2012-08-30 12:49:38,069::task::588::TaskManager.Task::(_updateState) Task=`2e7cf250-552b-41ce-9af7-d5c2730adfa0`::moving from state init -> state preparing >Thread-1621::INFO::2012-08-30 12:49:38,069::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1621::INFO::2012-08-30 12:49:38,069::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163388252258', 'lastCheck': 1346302169.3848281, 'code': 0, 'valid': True}} >Thread-1621::DEBUG::2012-08-30 12:49:38,070::task::1172::TaskManager.Task::(prepare) Task=`2e7cf250-552b-41ce-9af7-d5c2730adfa0`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163388252258', 'lastCheck': 1346302169.3848281, 'code': 0, 'valid': True}} >Thread-1621::DEBUG::2012-08-30 12:49:38,070::task::588::TaskManager.Task::(_updateState) Task=`2e7cf250-552b-41ce-9af7-d5c2730adfa0`::moving from state preparing -> state finished >Thread-1621::DEBUG::2012-08-30 12:49:38,070::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1621::DEBUG::2012-08-30 12:49:38,070::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1621::DEBUG::2012-08-30 12:49:38,071::task::978::TaskManager.Task::(_decref) Task=`2e7cf250-552b-41ce-9af7-d5c2730adfa0`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:49:40,082::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1622::DEBUG::2012-08-30 12:49:42,335::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1623::DEBUG::2012-08-30 12:49:42,335::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1622::DEBUG::2012-08-30 12:49:42,336::task::588::TaskManager.Task::(_updateState) Task=`10145ac5-3c4e-4c29-989b-892c00af43b8`::moving from state init -> state preparing >Thread-1623::DEBUG::2012-08-30 12:49:42,336::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1622::INFO::2012-08-30 12:49:42,336::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1622::INFO::2012-08-30 12:49:42,337::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1622::DEBUG::2012-08-30 12:49:42,338::task::1172::TaskManager.Task::(prepare) Task=`10145ac5-3c4e-4c29-989b-892c00af43b8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1622::DEBUG::2012-08-30 12:49:42,338::task::588::TaskManager.Task::(_updateState) Task=`10145ac5-3c4e-4c29-989b-892c00af43b8`::moving from state preparing -> state finished >Thread-1622::DEBUG::2012-08-30 12:49:42,338::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1622::DEBUG::2012-08-30 12:49:42,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1622::DEBUG::2012-08-30 12:49:42,339::task::978::TaskManager.Task::(_decref) Task=`10145ac5-3c4e-4c29-989b-892c00af43b8`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:49:43,025::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:45,084::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:45,084::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1624::DEBUG::2012-08-30 12:49:45,996::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1624::DEBUG::2012-08-30 12:49:45,996::task::588::TaskManager.Task::(_updateState) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::moving from state init -> state preparing >Thread-1624::INFO::2012-08-30 12:49:45,996::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1624::DEBUG::2012-08-30 12:49:45,997::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0524ec4b-e8ac-4e3a-be65-793a4fc37f5e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1624::DEBUG::2012-08-30 12:49:45,997::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1624::DEBUG::2012-08-30 12:49:45,997::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1624::DEBUG::2012-08-30 12:49:45,997::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0524ec4b-e8ac-4e3a-be65-793a4fc37f5e`::Granted request >Thread-1624::DEBUG::2012-08-30 12:49:45,998::task::817::TaskManager.Task::(resourceAcquired) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1624::DEBUG::2012-08-30 12:49:45,998::task::978::TaskManager.Task::(_decref) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::ref 1 aborting False >Thread-1624::INFO::2012-08-30 12:49:46,001::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1624::DEBUG::2012-08-30 12:49:46,001::task::1172::TaskManager.Task::(prepare) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1624::DEBUG::2012-08-30 12:49:46,001::task::588::TaskManager.Task::(_updateState) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::moving from state preparing -> state finished >Thread-1624::DEBUG::2012-08-30 12:49:46,002::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1624::DEBUG::2012-08-30 12:49:46,002::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1624::DEBUG::2012-08-30 12:49:46,002::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1624::DEBUG::2012-08-30 12:49:46,002::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1624::DEBUG::2012-08-30 12:49:46,003::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1624::DEBUG::2012-08-30 12:49:46,003::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1624::DEBUG::2012-08-30 12:49:46,003::task::978::TaskManager.Task::(_decref) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:49:48,026::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:49:50,085::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 12:49:53,027::task::588::TaskManager.Task::(_updateState) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:49:53,028::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:49:53,028::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c919ef2a-8971-4660-8335-eb74302d190b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:49:53,028::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:49:53,028::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:49:53,029::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c919ef2a-8971-4660-8335-eb74302d190b`::Granted request >Thread-40::DEBUG::2012-08-30 12:49:53,029::task::817::TaskManager.Task::(resourceAcquired) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:49:53,029::task::978::TaskManager.Task::(_decref) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:49:53,031::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:49:53,033::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:49:53,034::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:49:53,034::task::1172::TaskManager.Task::(prepare) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:49:53,035::task::588::TaskManager.Task::(_updateState) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:49:53,035::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:49:53,035::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:49:53,035::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:49:53,036::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:49:53,036::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:49:53,036::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:49:53,036::task::978::TaskManager.Task::(_decref) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:49:53,037::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:53,037::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:53,038::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:53,038::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:49:55,011::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:49:55,087::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:49:58,039::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1628::DEBUG::2012-08-30 12:49:59,214::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1628::DEBUG::2012-08-30 12:49:59,214::task::588::TaskManager.Task::(_updateState) Task=`0510cd18-91e1-4306-8417-c7ffaaa5475a`::moving from state init -> state preparing >Thread-1628::INFO::2012-08-30 12:49:59,214::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1628::INFO::2012-08-30 12:49:59,215::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1628::DEBUG::2012-08-30 12:49:59,215::task::1172::TaskManager.Task::(prepare) Task=`0510cd18-91e1-4306-8417-c7ffaaa5475a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1628::DEBUG::2012-08-30 12:49:59,215::task::588::TaskManager.Task::(_updateState) Task=`0510cd18-91e1-4306-8417-c7ffaaa5475a`::moving from state preparing -> state finished >Thread-1628::DEBUG::2012-08-30 12:49:59,215::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1628::DEBUG::2012-08-30 12:49:59,215::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1628::DEBUG::2012-08-30 12:49:59,216::task::978::TaskManager.Task::(_decref) Task=`0510cd18-91e1-4306-8417-c7ffaaa5475a`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:50:00,088::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:00,088::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:50:02,018::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1629::DEBUG::2012-08-30 12:50:02,402::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1629::DEBUG::2012-08-30 12:50:02,402::task::588::TaskManager.Task::(_updateState) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::moving from state init -> state preparing >Thread-1629::INFO::2012-08-30 12:50:02,402::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1629::DEBUG::2012-08-30 12:50:02,403::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`de30bf54-f9c0-4fb8-8b1c-a7b9b4a7fc0f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1629::DEBUG::2012-08-30 12:50:02,403::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1629::DEBUG::2012-08-30 12:50:02,404::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1629::DEBUG::2012-08-30 12:50:02,404::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`de30bf54-f9c0-4fb8-8b1c-a7b9b4a7fc0f`::Granted request >Thread-1629::DEBUG::2012-08-30 12:50:02,405::task::817::TaskManager.Task::(resourceAcquired) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1629::DEBUG::2012-08-30 12:50:02,405::task::978::TaskManager.Task::(_decref) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::ref 1 aborting False >Thread-1629::INFO::2012-08-30 12:50:02,408::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1629::DEBUG::2012-08-30 12:50:02,408::task::1172::TaskManager.Task::(prepare) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1629::DEBUG::2012-08-30 12:50:02,408::task::588::TaskManager.Task::(_updateState) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::moving from state preparing -> state finished >Thread-1629::DEBUG::2012-08-30 12:50:02,409::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1629::DEBUG::2012-08-30 12:50:02,409::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1629::DEBUG::2012-08-30 12:50:02,409::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1629::DEBUG::2012-08-30 12:50:02,409::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1629::DEBUG::2012-08-30 12:50:02,410::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1629::DEBUG::2012-08-30 12:50:02,410::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1629::DEBUG::2012-08-30 12:50:02,410::task::978::TaskManager.Task::(_decref) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:50:03,040::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:05,089::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1631::DEBUG::2012-08-30 12:50:07,584::task::588::TaskManager.Task::(_updateState) Task=`beb78f1e-de0e-435e-8a29-623ea8d18ec1`::moving from state init -> state preparing >Thread-1631::INFO::2012-08-30 12:50:07,584::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1631::INFO::2012-08-30 12:50:07,585::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177097320557', 'lastCheck': 1346302199.4052939, 'code': 0, 'valid': True}} >Thread-1631::DEBUG::2012-08-30 12:50:07,585::task::1172::TaskManager.Task::(prepare) Task=`beb78f1e-de0e-435e-8a29-623ea8d18ec1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177097320557', 'lastCheck': 1346302199.4052939, 'code': 0, 'valid': True}} >Thread-1631::DEBUG::2012-08-30 12:50:07,585::task::588::TaskManager.Task::(_updateState) Task=`beb78f1e-de0e-435e-8a29-623ea8d18ec1`::moving from state preparing -> state finished >Thread-1631::DEBUG::2012-08-30 12:50:07,585::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1631::DEBUG::2012-08-30 12:50:07,586::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1631::DEBUG::2012-08-30 12:50:07,586::task::978::TaskManager.Task::(_decref) Task=`beb78f1e-de0e-435e-8a29-623ea8d18ec1`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:50:08,041::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:08,041::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:10,090::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1632::DEBUG::2012-08-30 12:50:10,764::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1632::DEBUG::2012-08-30 12:50:10,764::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 12:50:13,042::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:15,091::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:15,092::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1633::DEBUG::2012-08-30 12:50:15,719::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1633::DEBUG::2012-08-30 12:50:15,719::task::588::TaskManager.Task::(_updateState) Task=`6015da28-ed4f-4f81-a716-cd8a60257d79`::moving from state init -> state preparing >Thread-1633::INFO::2012-08-30 12:50:15,719::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1633::INFO::2012-08-30 12:50:15,720::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1633::DEBUG::2012-08-30 12:50:15,720::task::1172::TaskManager.Task::(prepare) Task=`6015da28-ed4f-4f81-a716-cd8a60257d79`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1633::DEBUG::2012-08-30 12:50:15,720::task::588::TaskManager.Task::(_updateState) Task=`6015da28-ed4f-4f81-a716-cd8a60257d79`::moving from state preparing -> state finished >Thread-1633::DEBUG::2012-08-30 12:50:15,720::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1633::DEBUG::2012-08-30 12:50:15,721::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1633::DEBUG::2012-08-30 12:50:15,721::task::978::TaskManager.Task::(_decref) Task=`6015da28-ed4f-4f81-a716-cd8a60257d79`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:50:18,044::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1635::DEBUG::2012-08-30 12:50:18,918::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1635::DEBUG::2012-08-30 12:50:18,918::task::588::TaskManager.Task::(_updateState) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::moving from state init -> state preparing >Thread-1635::INFO::2012-08-30 12:50:18,919::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1635::DEBUG::2012-08-30 12:50:18,919::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1f1070f9-a5f0-41d2-93b2-860551c01d07`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1635::DEBUG::2012-08-30 12:50:18,919::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1635::DEBUG::2012-08-30 12:50:18,920::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1635::DEBUG::2012-08-30 12:50:18,920::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1f1070f9-a5f0-41d2-93b2-860551c01d07`::Granted request >Thread-1635::DEBUG::2012-08-30 12:50:18,920::task::817::TaskManager.Task::(resourceAcquired) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1635::DEBUG::2012-08-30 12:50:18,920::task::978::TaskManager.Task::(_decref) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::ref 1 aborting False >Thread-1635::INFO::2012-08-30 12:50:18,923::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1635::DEBUG::2012-08-30 12:50:18,924::task::1172::TaskManager.Task::(prepare) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1635::DEBUG::2012-08-30 12:50:18,924::task::588::TaskManager.Task::(_updateState) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::moving from state preparing -> state finished >Thread-1635::DEBUG::2012-08-30 12:50:18,924::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1635::DEBUG::2012-08-30 12:50:18,924::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1635::DEBUG::2012-08-30 12:50:18,925::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1635::DEBUG::2012-08-30 12:50:18,925::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1635::DEBUG::2012-08-30 12:50:18,925::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1635::DEBUG::2012-08-30 12:50:18,925::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1635::DEBUG::2012-08-30 12:50:18,926::task::978::TaskManager.Task::(_decref) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:50:20,092::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:23,045::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:23,046::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:50:25,039::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:50:25,093::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:28,047::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-26::DEBUG::2012-08-30 12:50:29,418::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 12:50:29,418::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-26::DEBUG::2012-08-30 12:50:29,419::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None) >Thread-26::DEBUG::2012-08-30 12:50:29,553::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5 >Thread-26::WARNING::2012-08-30 12:50:29,554::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found'] >Thread-26::DEBUG::2012-08-30 12:50:29,554::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-26::DEBUG::2012-08-30 12:50:29,559::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 12:50:29,560::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend >Thread-26::DEBUG::2012-08-30 12:50:29,562::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d'] >Thread-26::DEBUG::2012-08-30 12:50:29,563::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-26::WARNING::2012-08-30 12:50:29,564::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered >Thread-26::WARNING::2012-08-30 12:50:29,564::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered >Thread-104::DEBUG::2012-08-30 12:50:30,094::task::588::TaskManager.Task::(_updateState) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:50:30,095::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:50:30,095::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ac0ac618-5b41-43a0-82a4-43005baab9c4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:50:30,095::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:50:30,096::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:50:30,096::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ac0ac618-5b41-43a0-82a4-43005baab9c4`::Granted request >Thread-104::DEBUG::2012-08-30 12:50:30,096::task::817::TaskManager.Task::(resourceAcquired) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:50:30,097::task::978::TaskManager.Task::(_decref) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:50:30,098::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:50:30,100::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:50:30,101::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:50:30,102::task::1172::TaskManager.Task::(prepare) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:50:30,102::task::588::TaskManager.Task::(_updateState) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:50:30,102::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:50:30,102::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:50:30,103::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:50:30,103::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:50:30,103::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:50:30,103::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:50:30,104::task::978::TaskManager.Task::(_decref) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:50:30,104::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:30,104::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:30,105::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:30,105::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:50:32,046::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1639::DEBUG::2012-08-30 12:50:32,124::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1639::DEBUG::2012-08-30 12:50:32,125::task::588::TaskManager.Task::(_updateState) Task=`6887233b-e5d6-49b5-98cd-5a94956d479e`::moving from state init -> state preparing >Thread-1639::INFO::2012-08-30 12:50:32,125::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1639::INFO::2012-08-30 12:50:32,125::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1639::DEBUG::2012-08-30 12:50:32,125::task::1172::TaskManager.Task::(prepare) Task=`6887233b-e5d6-49b5-98cd-5a94956d479e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1639::DEBUG::2012-08-30 12:50:32,126::task::588::TaskManager.Task::(_updateState) Task=`6887233b-e5d6-49b5-98cd-5a94956d479e`::moving from state preparing -> state finished >Thread-1639::DEBUG::2012-08-30 12:50:32,126::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1639::DEBUG::2012-08-30 12:50:32,126::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1639::DEBUG::2012-08-30 12:50:32,126::task::978::TaskManager.Task::(_decref) Task=`6887233b-e5d6-49b5-98cd-5a94956d479e`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:50:33,048::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:35,106::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1645::DEBUG::2012-08-30 12:50:35,303::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1646::DEBUG::2012-08-30 12:50:35,304::task::588::TaskManager.Task::(_updateState) Task=`aab85601-58d4-431f-97c1-75afac3e4b13`::moving from state init -> state preparing >Thread-1645::DEBUG::2012-08-30 12:50:35,304::task::588::TaskManager.Task::(_updateState) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::moving from state init -> state preparing >Thread-1646::INFO::2012-08-30 12:50:35,305::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1645::INFO::2012-08-30 12:50:35,305::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1646::INFO::2012-08-30 12:50:35,305::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00158500671387', 'lastCheck': 1346302229.5707181, 'code': 0, 'valid': True}} >Thread-1645::DEBUG::2012-08-30 12:50:35,305::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`83062314-1940-4b2f-8dfa-f8d8be9a6d7e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1646::DEBUG::2012-08-30 12:50:35,306::task::1172::TaskManager.Task::(prepare) Task=`aab85601-58d4-431f-97c1-75afac3e4b13`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00158500671387', 'lastCheck': 1346302229.5707181, 'code': 0, 'valid': True}} >Thread-1645::DEBUG::2012-08-30 12:50:35,306::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1646::DEBUG::2012-08-30 12:50:35,306::task::588::TaskManager.Task::(_updateState) Task=`aab85601-58d4-431f-97c1-75afac3e4b13`::moving from state preparing -> state finished >Thread-1645::DEBUG::2012-08-30 12:50:35,306::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1646::DEBUG::2012-08-30 12:50:35,306::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1645::DEBUG::2012-08-30 12:50:35,307::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`83062314-1940-4b2f-8dfa-f8d8be9a6d7e`::Granted request >Thread-1646::DEBUG::2012-08-30 12:50:35,307::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1645::DEBUG::2012-08-30 12:50:35,307::task::817::TaskManager.Task::(resourceAcquired) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1646::DEBUG::2012-08-30 12:50:35,307::task::978::TaskManager.Task::(_decref) Task=`aab85601-58d4-431f-97c1-75afac3e4b13`::ref 0 aborting False >Thread-1645::DEBUG::2012-08-30 12:50:35,308::task::978::TaskManager.Task::(_decref) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::ref 1 aborting False >Thread-1645::INFO::2012-08-30 12:50:35,312::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1645::DEBUG::2012-08-30 12:50:35,313::task::1172::TaskManager.Task::(prepare) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1645::DEBUG::2012-08-30 12:50:35,313::task::588::TaskManager.Task::(_updateState) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::moving from state preparing -> state finished >Thread-1645::DEBUG::2012-08-30 12:50:35,313::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1645::DEBUG::2012-08-30 12:50:35,313::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1645::DEBUG::2012-08-30 12:50:35,314::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1645::DEBUG::2012-08-30 12:50:35,314::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1645::DEBUG::2012-08-30 12:50:35,314::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1645::DEBUG::2012-08-30 12:50:35,314::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1645::DEBUG::2012-08-30 12:50:35,315::task::978::TaskManager.Task::(_decref) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:50:38,049::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:38,050::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1647::DEBUG::2012-08-30 12:50:38,483::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1647::DEBUG::2012-08-30 12:50:38,483::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:50:40,107::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:43,051::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:45,108::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:50:45,109::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:48,052::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1649::DEBUG::2012-08-30 12:50:48,582::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1649::DEBUG::2012-08-30 12:50:48,583::task::588::TaskManager.Task::(_updateState) Task=`c0b99daf-1319-4d64-9902-578266598894`::moving from state init -> state preparing >Thread-1649::INFO::2012-08-30 12:50:48,583::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1649::INFO::2012-08-30 12:50:48,583::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1649::DEBUG::2012-08-30 12:50:48,583::task::1172::TaskManager.Task::(prepare) Task=`c0b99daf-1319-4d64-9902-578266598894`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1649::DEBUG::2012-08-30 12:50:48,584::task::588::TaskManager.Task::(_updateState) Task=`c0b99daf-1319-4d64-9902-578266598894`::moving from state preparing -> state finished >Thread-1649::DEBUG::2012-08-30 12:50:48,584::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1649::DEBUG::2012-08-30 12:50:48,584::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1649::DEBUG::2012-08-30 12:50:48,584::task::978::TaskManager.Task::(_decref) Task=`c0b99daf-1319-4d64-9902-578266598894`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:50:50,110::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1651::DEBUG::2012-08-30 12:50:51,780::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1651::DEBUG::2012-08-30 12:50:51,781::task::588::TaskManager.Task::(_updateState) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::moving from state init -> state preparing >Thread-1651::INFO::2012-08-30 12:50:51,781::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1651::DEBUG::2012-08-30 12:50:51,781::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1d29f26f-26f7-4b1b-8b3a-a52f5197583a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1651::DEBUG::2012-08-30 12:50:51,782::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1651::DEBUG::2012-08-30 12:50:51,782::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1651::DEBUG::2012-08-30 12:50:51,782::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1d29f26f-26f7-4b1b-8b3a-a52f5197583a`::Granted request >Thread-1651::DEBUG::2012-08-30 12:50:51,782::task::817::TaskManager.Task::(resourceAcquired) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1651::DEBUG::2012-08-30 12:50:51,783::task::978::TaskManager.Task::(_decref) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::ref 1 aborting False >Thread-1651::INFO::2012-08-30 12:50:51,786::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1651::DEBUG::2012-08-30 12:50:51,786::task::1172::TaskManager.Task::(prepare) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1651::DEBUG::2012-08-30 12:50:51,786::task::588::TaskManager.Task::(_updateState) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::moving from state preparing -> state finished >Thread-1651::DEBUG::2012-08-30 12:50:51,786::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1651::DEBUG::2012-08-30 12:50:51,786::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1651::DEBUG::2012-08-30 12:50:51,787::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1651::DEBUG::2012-08-30 12:50:51,787::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1651::DEBUG::2012-08-30 12:50:51,787::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1651::DEBUG::2012-08-30 12:50:51,787::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1651::DEBUG::2012-08-30 12:50:51,788::task::978::TaskManager.Task::(_decref) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 12:50:53,053::task::588::TaskManager.Task::(_updateState) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:50:53,054::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:50:53,054::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`18cfb057-2388-418e-907d-3c6b75587037`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:50:53,054::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:50:53,055::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:50:53,055::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`18cfb057-2388-418e-907d-3c6b75587037`::Granted request >Thread-40::DEBUG::2012-08-30 12:50:53,055::task::817::TaskManager.Task::(resourceAcquired) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:50:53,056::task::978::TaskManager.Task::(_decref) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:50:53,057::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:50:53,059::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:50:53,061::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:50:53,061::task::1172::TaskManager.Task::(prepare) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:50:53,061::task::588::TaskManager.Task::(_updateState) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:50:53,061::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:50:53,061::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:50:53,062::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:50:53,062::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:50:53,062::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:50:53,062::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:50:53,063::task::978::TaskManager.Task::(_decref) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:50:53,063::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:53,063::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:53,064::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:53,064::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:50:55,068::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:50:55,111::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:50:58,066::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:00,111::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:00,112::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:51:02,074::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:51:03,067::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1654::DEBUG::2012-08-30 12:51:03,438::task::588::TaskManager.Task::(_updateState) Task=`38f998ee-7b4d-41d8-addb-6e41bade9906`::moving from state init -> state preparing >Thread-1654::INFO::2012-08-30 12:51:03,438::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1654::INFO::2012-08-30 12:51:03,439::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177693367004', 'lastCheck': 1346302259.595521, 'code': 0, 'valid': True}} >Thread-1654::DEBUG::2012-08-30 12:51:03,439::task::1172::TaskManager.Task::(prepare) Task=`38f998ee-7b4d-41d8-addb-6e41bade9906`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177693367004', 'lastCheck': 1346302259.595521, 'code': 0, 'valid': True}} >Thread-1654::DEBUG::2012-08-30 12:51:03,439::task::588::TaskManager.Task::(_updateState) Task=`38f998ee-7b4d-41d8-addb-6e41bade9906`::moving from state preparing -> state finished >Thread-1654::DEBUG::2012-08-30 12:51:03,439::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1654::DEBUG::2012-08-30 12:51:03,439::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1654::DEBUG::2012-08-30 12:51:03,440::task::978::TaskManager.Task::(_decref) Task=`38f998ee-7b4d-41d8-addb-6e41bade9906`::ref 0 aborting False >Thread-1655::DEBUG::2012-08-30 12:51:05,031::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1655::DEBUG::2012-08-30 12:51:05,031::task::588::TaskManager.Task::(_updateState) Task=`41068544-9c30-4986-9b8f-bad4c6e13694`::moving from state init -> state preparing >Thread-1655::INFO::2012-08-30 12:51:05,031::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1655::INFO::2012-08-30 12:51:05,032::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1655::DEBUG::2012-08-30 12:51:05,032::task::1172::TaskManager.Task::(prepare) Task=`41068544-9c30-4986-9b8f-bad4c6e13694`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1655::DEBUG::2012-08-30 12:51:05,032::task::588::TaskManager.Task::(_updateState) Task=`41068544-9c30-4986-9b8f-bad4c6e13694`::moving from state preparing -> state finished >Thread-1655::DEBUG::2012-08-30 12:51:05,032::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1655::DEBUG::2012-08-30 12:51:05,033::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1655::DEBUG::2012-08-30 12:51:05,033::task::978::TaskManager.Task::(_decref) Task=`41068544-9c30-4986-9b8f-bad4c6e13694`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:51:05,113::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1656::DEBUG::2012-08-30 12:51:06,619::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1656::DEBUG::2012-08-30 12:51:06,620::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 12:51:08,068::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:08,069::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1657::DEBUG::2012-08-30 12:51:08,223::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1657::DEBUG::2012-08-30 12:51:08,223::task::588::TaskManager.Task::(_updateState) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::moving from state init -> state preparing >Thread-1657::INFO::2012-08-30 12:51:08,224::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1657::DEBUG::2012-08-30 12:51:08,224::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8e2f79e2-5000-4d08-9616-961ca5287152`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1657::DEBUG::2012-08-30 12:51:08,224::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1657::DEBUG::2012-08-30 12:51:08,225::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1657::DEBUG::2012-08-30 12:51:08,225::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8e2f79e2-5000-4d08-9616-961ca5287152`::Granted request >Thread-1657::DEBUG::2012-08-30 12:51:08,226::task::817::TaskManager.Task::(resourceAcquired) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1657::DEBUG::2012-08-30 12:51:08,226::task::978::TaskManager.Task::(_decref) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::ref 1 aborting False >Thread-1657::INFO::2012-08-30 12:51:08,229::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1657::DEBUG::2012-08-30 12:51:08,229::task::1172::TaskManager.Task::(prepare) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1657::DEBUG::2012-08-30 12:51:08,229::task::588::TaskManager.Task::(_updateState) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::moving from state preparing -> state finished >Thread-1657::DEBUG::2012-08-30 12:51:08,230::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1657::DEBUG::2012-08-30 12:51:08,230::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1657::DEBUG::2012-08-30 12:51:08,230::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1657::DEBUG::2012-08-30 12:51:08,230::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1657::DEBUG::2012-08-30 12:51:08,231::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1657::DEBUG::2012-08-30 12:51:08,231::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1657::DEBUG::2012-08-30 12:51:08,231::task::978::TaskManager.Task::(_decref) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:51:10,114::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:13,070::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:15,115::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:15,115::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:18,071::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:20,116::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1660::DEBUG::2012-08-30 12:51:21,441::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1660::DEBUG::2012-08-30 12:51:21,442::task::588::TaskManager.Task::(_updateState) Task=`12661056-e037-4b0a-8f19-8aba801ff02b`::moving from state init -> state preparing >Thread-1660::INFO::2012-08-30 12:51:21,442::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1660::INFO::2012-08-30 12:51:21,442::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1660::DEBUG::2012-08-30 12:51:21,442::task::1172::TaskManager.Task::(prepare) Task=`12661056-e037-4b0a-8f19-8aba801ff02b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1660::DEBUG::2012-08-30 12:51:21,443::task::588::TaskManager.Task::(_updateState) Task=`12661056-e037-4b0a-8f19-8aba801ff02b`::moving from state preparing -> state finished >Thread-1660::DEBUG::2012-08-30 12:51:21,443::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1660::DEBUG::2012-08-30 12:51:21,443::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1660::DEBUG::2012-08-30 12:51:21,443::task::978::TaskManager.Task::(_decref) Task=`12661056-e037-4b0a-8f19-8aba801ff02b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:51:23,073::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:23,073::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1662::DEBUG::2012-08-30 12:51:24,625::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1662::DEBUG::2012-08-30 12:51:24,626::task::588::TaskManager.Task::(_updateState) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::moving from state init -> state preparing >Thread-1662::INFO::2012-08-30 12:51:24,626::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1662::DEBUG::2012-08-30 12:51:24,626::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ed14358f-633e-4100-b8ad-5dbe6293f9fd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1662::DEBUG::2012-08-30 12:51:24,627::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1662::DEBUG::2012-08-30 12:51:24,627::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1662::DEBUG::2012-08-30 12:51:24,627::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ed14358f-633e-4100-b8ad-5dbe6293f9fd`::Granted request >Thread-1662::DEBUG::2012-08-30 12:51:24,627::task::817::TaskManager.Task::(resourceAcquired) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1662::DEBUG::2012-08-30 12:51:24,628::task::978::TaskManager.Task::(_decref) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::ref 1 aborting False >Thread-1662::INFO::2012-08-30 12:51:24,631::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1662::DEBUG::2012-08-30 12:51:24,631::task::1172::TaskManager.Task::(prepare) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1662::DEBUG::2012-08-30 12:51:24,631::task::588::TaskManager.Task::(_updateState) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::moving from state preparing -> state finished >Thread-1662::DEBUG::2012-08-30 12:51:24,631::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1662::DEBUG::2012-08-30 12:51:24,631::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1662::DEBUG::2012-08-30 12:51:24,632::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1662::DEBUG::2012-08-30 12:51:24,632::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1662::DEBUG::2012-08-30 12:51:24,632::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1662::DEBUG::2012-08-30 12:51:24,632::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1662::DEBUG::2012-08-30 12:51:24,633::task::978::TaskManager.Task::(_decref) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:51:25,094::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:51:25,117::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:28,074::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:51:30,118::task::588::TaskManager.Task::(_updateState) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:51:30,119::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:51:30,119::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`7b942bf1-9ea4-421e-b399-3ad6daeca5b7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:51:30,119::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:51:30,120::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:51:30,120::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`7b942bf1-9ea4-421e-b399-3ad6daeca5b7`::Granted request >Thread-104::DEBUG::2012-08-30 12:51:30,120::task::817::TaskManager.Task::(resourceAcquired) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:51:30,120::task::978::TaskManager.Task::(_decref) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:51:30,122::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:51:30,124::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:51:30,125::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:51:30,125::task::1172::TaskManager.Task::(prepare) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:51:30,125::task::588::TaskManager.Task::(_updateState) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:51:30,126::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:51:30,126::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:51:30,126::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:51:30,126::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:51:30,127::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:51:30,127::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:51:30,127::task::978::TaskManager.Task::(_decref) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:51:30,128::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:30,128::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:30,128::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:30,129::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:51:32,100::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1664::DEBUG::2012-08-30 12:51:32,735::task::588::TaskManager.Task::(_updateState) Task=`39e366dc-6ee4-4c35-9974-ea796d8c2904`::moving from state init -> state preparing >Thread-1664::INFO::2012-08-30 12:51:32,736::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1664::INFO::2012-08-30 12:51:32,736::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346302289.6153109, 'code': 0, 'valid': True}} >Thread-1664::DEBUG::2012-08-30 12:51:32,736::task::1172::TaskManager.Task::(prepare) Task=`39e366dc-6ee4-4c35-9974-ea796d8c2904`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346302289.6153109, 'code': 0, 'valid': True}} >Thread-1664::DEBUG::2012-08-30 12:51:32,737::task::588::TaskManager.Task::(_updateState) Task=`39e366dc-6ee4-4c35-9974-ea796d8c2904`::moving from state preparing -> state finished >Thread-1664::DEBUG::2012-08-30 12:51:32,737::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1664::DEBUG::2012-08-30 12:51:32,737::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1664::DEBUG::2012-08-30 12:51:32,737::task::978::TaskManager.Task::(_decref) Task=`39e366dc-6ee4-4c35-9974-ea796d8c2904`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:51:33,076::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:35,130::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1665::DEBUG::2012-08-30 12:51:36,205::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1665::DEBUG::2012-08-30 12:51:36,206::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1666::DEBUG::2012-08-30 12:51:37,797::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1666::DEBUG::2012-08-30 12:51:37,798::task::588::TaskManager.Task::(_updateState) Task=`3e8f7ca4-e61b-4dfc-8a60-5246274228a2`::moving from state init -> state preparing >Thread-1666::INFO::2012-08-30 12:51:37,798::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1666::INFO::2012-08-30 12:51:37,798::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1666::DEBUG::2012-08-30 12:51:37,798::task::1172::TaskManager.Task::(prepare) Task=`3e8f7ca4-e61b-4dfc-8a60-5246274228a2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1666::DEBUG::2012-08-30 12:51:37,799::task::588::TaskManager.Task::(_updateState) Task=`3e8f7ca4-e61b-4dfc-8a60-5246274228a2`::moving from state preparing -> state finished >Thread-1666::DEBUG::2012-08-30 12:51:37,799::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1666::DEBUG::2012-08-30 12:51:37,799::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1666::DEBUG::2012-08-30 12:51:37,799::task::978::TaskManager.Task::(_decref) Task=`3e8f7ca4-e61b-4dfc-8a60-5246274228a2`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:51:38,077::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:38,078::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:40,131::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1667::DEBUG::2012-08-30 12:51:41,013::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1667::DEBUG::2012-08-30 12:51:41,014::task::588::TaskManager.Task::(_updateState) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::moving from state init -> state preparing >Thread-1667::INFO::2012-08-30 12:51:41,014::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1667::DEBUG::2012-08-30 12:51:41,015::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3a0552d9-a90a-4c12-a765-81ef7ae07c99`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1667::DEBUG::2012-08-30 12:51:41,015::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1667::DEBUG::2012-08-30 12:51:41,015::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1667::DEBUG::2012-08-30 12:51:41,016::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3a0552d9-a90a-4c12-a765-81ef7ae07c99`::Granted request >Thread-1667::DEBUG::2012-08-30 12:51:41,016::task::817::TaskManager.Task::(resourceAcquired) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1667::DEBUG::2012-08-30 12:51:41,016::task::978::TaskManager.Task::(_decref) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::ref 1 aborting False >Thread-1667::INFO::2012-08-30 12:51:41,019::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1667::DEBUG::2012-08-30 12:51:41,019::task::1172::TaskManager.Task::(prepare) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1667::DEBUG::2012-08-30 12:51:41,020::task::588::TaskManager.Task::(_updateState) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::moving from state preparing -> state finished >Thread-1667::DEBUG::2012-08-30 12:51:41,020::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1667::DEBUG::2012-08-30 12:51:41,020::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1667::DEBUG::2012-08-30 12:51:41,020::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1667::DEBUG::2012-08-30 12:51:41,021::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1667::DEBUG::2012-08-30 12:51:41,021::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1667::DEBUG::2012-08-30 12:51:41,021::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1667::DEBUG::2012-08-30 12:51:41,021::task::978::TaskManager.Task::(_decref) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:51:43,079::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:45,132::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:45,132::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:48,080::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:51:50,133::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 12:51:53,081::task::588::TaskManager.Task::(_updateState) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:51:53,082::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:51:53,082::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9307a3e0-c66c-4031-a41f-78b3a4e56bf3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:51:53,082::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:51:53,083::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:51:53,083::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9307a3e0-c66c-4031-a41f-78b3a4e56bf3`::Granted request >Thread-40::DEBUG::2012-08-30 12:51:53,083::task::817::TaskManager.Task::(resourceAcquired) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:51:53,084::task::978::TaskManager.Task::(_decref) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:51:53,086::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:51:53,088::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:51:53,090::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:51:53,090::task::1172::TaskManager.Task::(prepare) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:51:53,090::task::588::TaskManager.Task::(_updateState) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:51:53,090::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:51:53,090::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:51:53,091::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:51:53,091::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:51:53,091::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:51:53,091::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:51:53,092::task::978::TaskManager.Task::(_decref) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:51:53,092::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:53,093::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:53,093::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:51:53,093::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1671::DEBUG::2012-08-30 12:51:54,231::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1671::DEBUG::2012-08-30 12:51:54,232::task::588::TaskManager.Task::(_updateState) Task=`395563fb-8486-471f-a4f6-1dadc9446a18`::moving from state init -> state preparing >Thread-1671::INFO::2012-08-30 12:51:54,232::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1671::INFO::2012-08-30 12:51:54,232::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1671::DEBUG::2012-08-30 12:51:54,233::task::1172::TaskManager.Task::(prepare) Task=`395563fb-8486-471f-a4f6-1dadc9446a18`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1671::DEBUG::2012-08-30 12:51:54,233::task::588::TaskManager.Task::(_updateState) Task=`395563fb-8486-471f-a4f6-1dadc9446a18`::moving from state preparing -> state finished >Thread-1671::DEBUG::2012-08-30 12:51:54,233::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1671::DEBUG::2012-08-30 12:51:54,233::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1671::DEBUG::2012-08-30 12:51:54,234::task::978::TaskManager.Task::(_decref) Task=`395563fb-8486-471f-a4f6-1dadc9446a18`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:51:55,120::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-104::ERROR::2012-08-30 12:51:55,134::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1673::DEBUG::2012-08-30 12:51:57,441::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1673::DEBUG::2012-08-30 12:51:57,441::task::588::TaskManager.Task::(_updateState) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::moving from state init -> state preparing >Thread-1673::INFO::2012-08-30 12:51:57,441::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1673::DEBUG::2012-08-30 12:51:57,442::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f8017968-c826-417c-b4d2-fc81cdcf2490`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1673::DEBUG::2012-08-30 12:51:57,442::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1673::DEBUG::2012-08-30 12:51:57,442::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1673::DEBUG::2012-08-30 12:51:57,443::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f8017968-c826-417c-b4d2-fc81cdcf2490`::Granted request >Thread-1673::DEBUG::2012-08-30 12:51:57,443::task::817::TaskManager.Task::(resourceAcquired) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1673::DEBUG::2012-08-30 12:51:57,443::task::978::TaskManager.Task::(_decref) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::ref 1 aborting False >Thread-1673::INFO::2012-08-30 12:51:57,446::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1673::DEBUG::2012-08-30 12:51:57,446::task::1172::TaskManager.Task::(prepare) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1673::DEBUG::2012-08-30 12:51:57,447::task::588::TaskManager.Task::(_updateState) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::moving from state preparing -> state finished >Thread-1673::DEBUG::2012-08-30 12:51:57,447::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1673::DEBUG::2012-08-30 12:51:57,447::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1673::DEBUG::2012-08-30 12:51:57,447::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1673::DEBUG::2012-08-30 12:51:57,448::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1673::DEBUG::2012-08-30 12:51:57,448::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1673::DEBUG::2012-08-30 12:51:57,448::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1673::DEBUG::2012-08-30 12:51:57,448::task::978::TaskManager.Task::(_decref) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:51:58,095::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:00,135::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:00,135::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1674::DEBUG::2012-08-30 12:52:01,783::task::588::TaskManager.Task::(_updateState) Task=`0ff8264a-ed9f-443c-9296-ff1323a963dd`::moving from state init -> state preparing >Thread-1674::INFO::2012-08-30 12:52:01,783::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1674::INFO::2012-08-30 12:52:01,784::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177407264709', 'lastCheck': 1346302319.635859, 'code': 0, 'valid': True}} >Thread-1674::DEBUG::2012-08-30 12:52:01,784::task::1172::TaskManager.Task::(prepare) Task=`0ff8264a-ed9f-443c-9296-ff1323a963dd`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177407264709', 'lastCheck': 1346302319.635859, 'code': 0, 'valid': True}} >Thread-1674::DEBUG::2012-08-30 12:52:01,784::task::588::TaskManager.Task::(_updateState) Task=`0ff8264a-ed9f-443c-9296-ff1323a963dd`::moving from state preparing -> state finished >Thread-1674::DEBUG::2012-08-30 12:52:01,784::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1674::DEBUG::2012-08-30 12:52:01,785::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1674::DEBUG::2012-08-30 12:52:01,785::task::978::TaskManager.Task::(_decref) Task=`0ff8264a-ed9f-443c-9296-ff1323a963dd`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:52:02,127::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:52:03,096::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1675::DEBUG::2012-08-30 12:52:05,013::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1675::DEBUG::2012-08-30 12:52:05,014::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:52:05,136::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:08,097::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:08,098::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:10,137::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1677::DEBUG::2012-08-30 12:52:10,314::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1677::DEBUG::2012-08-30 12:52:10,315::task::588::TaskManager.Task::(_updateState) Task=`185a03ad-12dd-4f38-9a58-c536b559ee29`::moving from state init -> state preparing >Thread-1677::INFO::2012-08-30 12:52:10,315::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1677::INFO::2012-08-30 12:52:10,315::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1677::DEBUG::2012-08-30 12:52:10,316::task::1172::TaskManager.Task::(prepare) Task=`185a03ad-12dd-4f38-9a58-c536b559ee29`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1677::DEBUG::2012-08-30 12:52:10,316::task::588::TaskManager.Task::(_updateState) Task=`185a03ad-12dd-4f38-9a58-c536b559ee29`::moving from state preparing -> state finished >Thread-1677::DEBUG::2012-08-30 12:52:10,316::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1677::DEBUG::2012-08-30 12:52:10,317::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1677::DEBUG::2012-08-30 12:52:10,317::task::978::TaskManager.Task::(_decref) Task=`185a03ad-12dd-4f38-9a58-c536b559ee29`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:52:13,099::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1678::DEBUG::2012-08-30 12:52:13,837::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1678::DEBUG::2012-08-30 12:52:13,837::task::588::TaskManager.Task::(_updateState) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::moving from state init -> state preparing >Thread-1678::INFO::2012-08-30 12:52:13,838::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1678::DEBUG::2012-08-30 12:52:13,838::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d11ffc6e-744d-4863-95c0-265aa80a2b88`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1678::DEBUG::2012-08-30 12:52:13,838::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1678::DEBUG::2012-08-30 12:52:13,838::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1678::DEBUG::2012-08-30 12:52:13,839::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d11ffc6e-744d-4863-95c0-265aa80a2b88`::Granted request >Thread-1678::DEBUG::2012-08-30 12:52:13,839::task::817::TaskManager.Task::(resourceAcquired) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1678::DEBUG::2012-08-30 12:52:13,839::task::978::TaskManager.Task::(_decref) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::ref 1 aborting False >Thread-1678::INFO::2012-08-30 12:52:13,842::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1678::DEBUG::2012-08-30 12:52:13,842::task::1172::TaskManager.Task::(prepare) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1678::DEBUG::2012-08-30 12:52:13,843::task::588::TaskManager.Task::(_updateState) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::moving from state preparing -> state finished >Thread-1678::DEBUG::2012-08-30 12:52:13,843::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1678::DEBUG::2012-08-30 12:52:13,843::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1678::DEBUG::2012-08-30 12:52:13,843::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1678::DEBUG::2012-08-30 12:52:13,844::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1678::DEBUG::2012-08-30 12:52:13,844::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1678::DEBUG::2012-08-30 12:52:13,844::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1678::DEBUG::2012-08-30 12:52:13,844::task::978::TaskManager.Task::(_decref) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:52:15,138::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:15,139::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:18,100::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:20,140::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:23,102::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:23,102::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:25,140::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:52:25,144::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1682::DEBUG::2012-08-30 12:52:27,057::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1682::DEBUG::2012-08-30 12:52:27,058::task::588::TaskManager.Task::(_updateState) Task=`a422784f-d661-493e-bda3-217b760138ad`::moving from state init -> state preparing >Thread-1682::INFO::2012-08-30 12:52:27,058::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1682::INFO::2012-08-30 12:52:27,058::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1682::DEBUG::2012-08-30 12:52:27,059::task::1172::TaskManager.Task::(prepare) Task=`a422784f-d661-493e-bda3-217b760138ad`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1682::DEBUG::2012-08-30 12:52:27,059::task::588::TaskManager.Task::(_updateState) Task=`a422784f-d661-493e-bda3-217b760138ad`::moving from state preparing -> state finished >Thread-1682::DEBUG::2012-08-30 12:52:27,059::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1682::DEBUG::2012-08-30 12:52:27,059::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1682::DEBUG::2012-08-30 12:52:27,059::task::978::TaskManager.Task::(_decref) Task=`a422784f-d661-493e-bda3-217b760138ad`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:52:28,103::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:52:30,141::task::588::TaskManager.Task::(_updateState) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:52:30,142::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:52:30,142::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`94378283-af42-4b94-ba7e-8198df2ace1b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:52:30,142::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:52:30,143::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:52:30,143::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`94378283-af42-4b94-ba7e-8198df2ace1b`::Granted request >Thread-104::DEBUG::2012-08-30 12:52:30,143::task::817::TaskManager.Task::(resourceAcquired) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:52:30,143::task::978::TaskManager.Task::(_decref) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:52:30,145::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:52:30,147::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:52:30,148::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:52:30,148::task::1172::TaskManager.Task::(prepare) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:52:30,149::task::588::TaskManager.Task::(_updateState) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:52:30,149::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:52:30,149::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:52:30,149::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:52:30,150::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:52:30,150::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:52:30,150::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:52:30,150::task::978::TaskManager.Task::(_decref) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:52:30,151::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:30,151::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:30,151::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:30,152::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1683::DEBUG::2012-08-30 12:52:30,242::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1684::DEBUG::2012-08-30 12:52:30,243::task::588::TaskManager.Task::(_updateState) Task=`6804587e-fbc2-4bbf-b9ae-3ba82fdab845`::moving from state init -> state preparing >Thread-1683::DEBUG::2012-08-30 12:52:30,243::task::588::TaskManager.Task::(_updateState) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::moving from state init -> state preparing >Thread-1684::INFO::2012-08-30 12:52:30,243::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1683::INFO::2012-08-30 12:52:30,243::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1684::INFO::2012-08-30 12:52:30,243::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161409378052', 'lastCheck': 1346302349.654927, 'code': 0, 'valid': True}} >Thread-1683::DEBUG::2012-08-30 12:52:30,244::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f01980df-e962-4791-814a-b3a6045de8c7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1684::DEBUG::2012-08-30 12:52:30,244::task::1172::TaskManager.Task::(prepare) Task=`6804587e-fbc2-4bbf-b9ae-3ba82fdab845`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161409378052', 'lastCheck': 1346302349.654927, 'code': 0, 'valid': True}} >Thread-1683::DEBUG::2012-08-30 12:52:30,244::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1684::DEBUG::2012-08-30 12:52:30,244::task::588::TaskManager.Task::(_updateState) Task=`6804587e-fbc2-4bbf-b9ae-3ba82fdab845`::moving from state preparing -> state finished >Thread-1683::DEBUG::2012-08-30 12:52:30,245::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1684::DEBUG::2012-08-30 12:52:30,245::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1683::DEBUG::2012-08-30 12:52:30,245::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f01980df-e962-4791-814a-b3a6045de8c7`::Granted request >Thread-1684::DEBUG::2012-08-30 12:52:30,245::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1683::DEBUG::2012-08-30 12:52:30,246::task::817::TaskManager.Task::(resourceAcquired) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1684::DEBUG::2012-08-30 12:52:30,246::task::978::TaskManager.Task::(_decref) Task=`6804587e-fbc2-4bbf-b9ae-3ba82fdab845`::ref 0 aborting False >Thread-1683::DEBUG::2012-08-30 12:52:30,246::task::978::TaskManager.Task::(_decref) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::ref 1 aborting False >Thread-1683::INFO::2012-08-30 12:52:30,251::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1683::DEBUG::2012-08-30 12:52:30,251::task::1172::TaskManager.Task::(prepare) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1683::DEBUG::2012-08-30 12:52:30,251::task::588::TaskManager.Task::(_updateState) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::moving from state preparing -> state finished >Thread-1683::DEBUG::2012-08-30 12:52:30,252::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1683::DEBUG::2012-08-30 12:52:30,252::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1683::DEBUG::2012-08-30 12:52:30,252::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1683::DEBUG::2012-08-30 12:52:30,252::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1683::DEBUG::2012-08-30 12:52:30,253::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1683::DEBUG::2012-08-30 12:52:30,253::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1683::DEBUG::2012-08-30 12:52:30,253::task::978::TaskManager.Task::(_decref) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:52:32,150::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:52:33,105::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1685::DEBUG::2012-08-30 12:52:33,451::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1685::DEBUG::2012-08-30 12:52:33,451::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:52:35,153::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:38,106::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:38,106::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:40,154::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:43,107::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1687::DEBUG::2012-08-30 12:52:43,494::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1687::DEBUG::2012-08-30 12:52:43,495::task::588::TaskManager.Task::(_updateState) Task=`b4d19073-8cda-4160-9240-6d05063162c8`::moving from state init -> state preparing >Thread-1687::INFO::2012-08-30 12:52:43,496::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1687::INFO::2012-08-30 12:52:43,496::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1687::DEBUG::2012-08-30 12:52:43,496::task::1172::TaskManager.Task::(prepare) Task=`b4d19073-8cda-4160-9240-6d05063162c8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1687::DEBUG::2012-08-30 12:52:43,496::task::588::TaskManager.Task::(_updateState) Task=`b4d19073-8cda-4160-9240-6d05063162c8`::moving from state preparing -> state finished >Thread-1687::DEBUG::2012-08-30 12:52:43,497::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1687::DEBUG::2012-08-30 12:52:43,497::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1687::DEBUG::2012-08-30 12:52:43,497::task::978::TaskManager.Task::(_decref) Task=`b4d19073-8cda-4160-9240-6d05063162c8`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:52:45,155::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:45,155::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1689::DEBUG::2012-08-30 12:52:46,681::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1689::DEBUG::2012-08-30 12:52:46,681::task::588::TaskManager.Task::(_updateState) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::moving from state init -> state preparing >Thread-1689::INFO::2012-08-30 12:52:46,681::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1689::DEBUG::2012-08-30 12:52:46,682::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9d7d506b-600d-4283-8e41-75e8322565e2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1689::DEBUG::2012-08-30 12:52:46,682::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1689::DEBUG::2012-08-30 12:52:46,682::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1689::DEBUG::2012-08-30 12:52:46,682::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9d7d506b-600d-4283-8e41-75e8322565e2`::Granted request >Thread-1689::DEBUG::2012-08-30 12:52:46,683::task::817::TaskManager.Task::(resourceAcquired) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1689::DEBUG::2012-08-30 12:52:46,683::task::978::TaskManager.Task::(_decref) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::ref 1 aborting False >Thread-1689::INFO::2012-08-30 12:52:46,686::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1689::DEBUG::2012-08-30 12:52:46,686::task::1172::TaskManager.Task::(prepare) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1689::DEBUG::2012-08-30 12:52:46,686::task::588::TaskManager.Task::(_updateState) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::moving from state preparing -> state finished >Thread-1689::DEBUG::2012-08-30 12:52:46,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1689::DEBUG::2012-08-30 12:52:46,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1689::DEBUG::2012-08-30 12:52:46,687::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1689::DEBUG::2012-08-30 12:52:46,687::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1689::DEBUG::2012-08-30 12:52:46,688::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1689::DEBUG::2012-08-30 12:52:46,688::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1689::DEBUG::2012-08-30 12:52:46,688::task::978::TaskManager.Task::(_decref) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:52:48,111::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:50,156::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 12:52:53,113::task::588::TaskManager.Task::(_updateState) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:52:53,113::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:52:53,114::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`76581dbc-e7d3-4998-8fa8-7d4fbb49ec2c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:52:53,114::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:52:53,114::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:52:53,114::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`76581dbc-e7d3-4998-8fa8-7d4fbb49ec2c`::Granted request >Thread-40::DEBUG::2012-08-30 12:52:53,115::task::817::TaskManager.Task::(resourceAcquired) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:52:53,115::task::978::TaskManager.Task::(_decref) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:52:53,117::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:52:53,119::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:52:53,120::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:52:53,120::task::1172::TaskManager.Task::(prepare) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:52:53,120::task::588::TaskManager.Task::(_updateState) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:52:53,121::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:52:53,121::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:52:53,121::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:52:53,121::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:52:53,122::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:52:53,122::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:52:53,122::task::978::TaskManager.Task::(_decref) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:52:53,122::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:53,123::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:53,123::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:52:53,124::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:52:55,157::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:52:55,166::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 12:52:58,125::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1692::DEBUG::2012-08-30 12:52:58,314::task::588::TaskManager.Task::(_updateState) Task=`44d12fb8-aab8-49da-87af-a857e29015e6`::moving from state init -> state preparing >Thread-1692::INFO::2012-08-30 12:52:58,314::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1692::INFO::2012-08-30 12:52:58,314::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177884101868', 'lastCheck': 1346302369.668272, 'code': 0, 'valid': True}} >Thread-1692::DEBUG::2012-08-30 12:52:58,315::task::1172::TaskManager.Task::(prepare) Task=`44d12fb8-aab8-49da-87af-a857e29015e6`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177884101868', 'lastCheck': 1346302369.668272, 'code': 0, 'valid': True}} >Thread-1692::DEBUG::2012-08-30 12:52:58,315::task::588::TaskManager.Task::(_updateState) Task=`44d12fb8-aab8-49da-87af-a857e29015e6`::moving from state preparing -> state finished >Thread-1692::DEBUG::2012-08-30 12:52:58,315::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1692::DEBUG::2012-08-30 12:52:58,315::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1692::DEBUG::2012-08-30 12:52:58,316::task::978::TaskManager.Task::(_decref) Task=`44d12fb8-aab8-49da-87af-a857e29015e6`::ref 0 aborting False >Thread-1693::DEBUG::2012-08-30 12:52:59,897::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1693::DEBUG::2012-08-30 12:52:59,898::task::588::TaskManager.Task::(_updateState) Task=`fa25be25-00ce-49c9-aa14-76bf02f4f618`::moving from state init -> state preparing >Thread-1693::INFO::2012-08-30 12:52:59,898::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1693::INFO::2012-08-30 12:52:59,898::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1693::DEBUG::2012-08-30 12:52:59,899::task::1172::TaskManager.Task::(prepare) Task=`fa25be25-00ce-49c9-aa14-76bf02f4f618`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1693::DEBUG::2012-08-30 12:52:59,899::task::588::TaskManager.Task::(_updateState) Task=`fa25be25-00ce-49c9-aa14-76bf02f4f618`::moving from state preparing -> state finished >Thread-1693::DEBUG::2012-08-30 12:52:59,899::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1693::DEBUG::2012-08-30 12:52:59,899::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1693::DEBUG::2012-08-30 12:52:59,900::task::978::TaskManager.Task::(_decref) Task=`fa25be25-00ce-49c9-aa14-76bf02f4f618`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:53:00,158::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:00,159::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1694::DEBUG::2012-08-30 12:53:01,535::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1694::DEBUG::2012-08-30 12:53:01,535::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >VM Channels Listener::DEBUG::2012-08-30 12:53:02,171::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1695::DEBUG::2012-08-30 12:53:03,126::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-40::ERROR::2012-08-30 12:53:03,126::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1695::DEBUG::2012-08-30 12:53:03,126::task::588::TaskManager.Task::(_updateState) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::moving from state init -> state preparing >Thread-1695::INFO::2012-08-30 12:53:03,127::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1695::DEBUG::2012-08-30 12:53:03,127::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`fa469b9a-d82c-4631-9a86-934fb4409847`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1695::DEBUG::2012-08-30 12:53:03,127::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1695::DEBUG::2012-08-30 12:53:03,128::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1695::DEBUG::2012-08-30 12:53:03,128::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`fa469b9a-d82c-4631-9a86-934fb4409847`::Granted request >Thread-1695::DEBUG::2012-08-30 12:53:03,128::task::817::TaskManager.Task::(resourceAcquired) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1695::DEBUG::2012-08-30 12:53:03,128::task::978::TaskManager.Task::(_decref) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::ref 1 aborting False >Thread-1695::INFO::2012-08-30 12:53:03,131::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1695::DEBUG::2012-08-30 12:53:03,132::task::1172::TaskManager.Task::(prepare) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1695::DEBUG::2012-08-30 12:53:03,132::task::588::TaskManager.Task::(_updateState) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::moving from state preparing -> state finished >Thread-1695::DEBUG::2012-08-30 12:53:03,132::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1695::DEBUG::2012-08-30 12:53:03,132::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1695::DEBUG::2012-08-30 12:53:03,133::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1695::DEBUG::2012-08-30 12:53:03,133::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1695::DEBUG::2012-08-30 12:53:03,133::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1695::DEBUG::2012-08-30 12:53:03,133::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1695::DEBUG::2012-08-30 12:53:03,134::task::978::TaskManager.Task::(_decref) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:53:05,159::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:08,127::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:08,128::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:10,160::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:13,129::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:15,161::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:15,162::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1698::DEBUG::2012-08-30 12:53:16,340::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1698::DEBUG::2012-08-30 12:53:16,340::task::588::TaskManager.Task::(_updateState) Task=`387fe826-f739-42af-9ad9-5b6b94790202`::moving from state init -> state preparing >Thread-1698::INFO::2012-08-30 12:53:16,340::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1698::INFO::2012-08-30 12:53:16,341::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1698::DEBUG::2012-08-30 12:53:16,341::task::1172::TaskManager.Task::(prepare) Task=`387fe826-f739-42af-9ad9-5b6b94790202`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1698::DEBUG::2012-08-30 12:53:16,341::task::588::TaskManager.Task::(_updateState) Task=`387fe826-f739-42af-9ad9-5b6b94790202`::moving from state preparing -> state finished >Thread-1698::DEBUG::2012-08-30 12:53:16,341::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1698::DEBUG::2012-08-30 12:53:16,342::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1698::DEBUG::2012-08-30 12:53:16,342::task::978::TaskManager.Task::(_decref) Task=`387fe826-f739-42af-9ad9-5b6b94790202`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:53:18,130::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1700::DEBUG::2012-08-30 12:53:19,515::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1700::DEBUG::2012-08-30 12:53:19,515::task::588::TaskManager.Task::(_updateState) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::moving from state init -> state preparing >Thread-1700::INFO::2012-08-30 12:53:19,515::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1700::DEBUG::2012-08-30 12:53:19,516::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d6f1f642-d46e-4df5-b13e-19fe5e6fe111`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1700::DEBUG::2012-08-30 12:53:19,516::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1700::DEBUG::2012-08-30 12:53:19,516::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1700::DEBUG::2012-08-30 12:53:19,517::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d6f1f642-d46e-4df5-b13e-19fe5e6fe111`::Granted request >Thread-1700::DEBUG::2012-08-30 12:53:19,517::task::817::TaskManager.Task::(resourceAcquired) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1700::DEBUG::2012-08-30 12:53:19,517::task::978::TaskManager.Task::(_decref) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::ref 1 aborting False >Thread-1700::INFO::2012-08-30 12:53:19,520::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1700::DEBUG::2012-08-30 12:53:19,520::task::1172::TaskManager.Task::(prepare) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1700::DEBUG::2012-08-30 12:53:19,521::task::588::TaskManager.Task::(_updateState) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::moving from state preparing -> state finished >Thread-1700::DEBUG::2012-08-30 12:53:19,521::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1700::DEBUG::2012-08-30 12:53:19,521::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1700::DEBUG::2012-08-30 12:53:19,521::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1700::DEBUG::2012-08-30 12:53:19,522::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1700::DEBUG::2012-08-30 12:53:19,522::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1700::DEBUG::2012-08-30 12:53:19,522::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1700::DEBUG::2012-08-30 12:53:19,522::task::978::TaskManager.Task::(_decref) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:53:20,163::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:23,132::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:23,132::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:25,164::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:53:25,191::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1702::DEBUG::2012-08-30 12:53:27,553::task::588::TaskManager.Task::(_updateState) Task=`c62166f2-de18-47b0-b9b6-3d26fe63ab29`::moving from state init -> state preparing >Thread-1702::INFO::2012-08-30 12:53:27,553::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1702::INFO::2012-08-30 12:53:27,553::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016348361969', 'lastCheck': 1346302399.6880889, 'code': 0, 'valid': True}} >Thread-1702::DEBUG::2012-08-30 12:53:27,554::task::1172::TaskManager.Task::(prepare) Task=`c62166f2-de18-47b0-b9b6-3d26fe63ab29`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016348361969', 'lastCheck': 1346302399.6880889, 'code': 0, 'valid': True}} >Thread-1702::DEBUG::2012-08-30 12:53:27,554::task::588::TaskManager.Task::(_updateState) Task=`c62166f2-de18-47b0-b9b6-3d26fe63ab29`::moving from state preparing -> state finished >Thread-1702::DEBUG::2012-08-30 12:53:27,554::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1702::DEBUG::2012-08-30 12:53:27,554::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1702::DEBUG::2012-08-30 12:53:27,554::task::978::TaskManager.Task::(_decref) Task=`c62166f2-de18-47b0-b9b6-3d26fe63ab29`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:53:28,133::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:53:30,165::task::588::TaskManager.Task::(_updateState) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:53:30,165::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:53:30,165::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bbbff8cd-08ce-4e81-a77c-1a3f38a80dbe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:53:30,166::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:53:30,166::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:53:30,166::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bbbff8cd-08ce-4e81-a77c-1a3f38a80dbe`::Granted request >Thread-104::DEBUG::2012-08-30 12:53:30,166::task::817::TaskManager.Task::(resourceAcquired) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:53:30,167::task::978::TaskManager.Task::(_decref) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:53:30,168::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:53:30,170::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:53:30,171::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:53:30,172::task::1172::TaskManager.Task::(prepare) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:53:30,172::task::588::TaskManager.Task::(_updateState) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:53:30,172::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:53:30,172::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:53:30,173::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:53:30,173::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:53:30,173::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:53:30,173::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:53:30,173::task::978::TaskManager.Task::(_decref) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:53:30,174::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:30,174::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:30,175::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:30,175::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1703::DEBUG::2012-08-30 12:53:30,743::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1703::DEBUG::2012-08-30 12:53:30,744::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >VM Channels Listener::DEBUG::2012-08-30 12:53:32,198::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1704::DEBUG::2012-08-30 12:53:32,329::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1704::DEBUG::2012-08-30 12:53:32,329::task::588::TaskManager.Task::(_updateState) Task=`e9e9e3f5-95a0-42bd-b277-c8d624eb415c`::moving from state init -> state preparing >Thread-1704::INFO::2012-08-30 12:53:32,329::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1704::INFO::2012-08-30 12:53:32,329::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1704::DEBUG::2012-08-30 12:53:32,330::task::1172::TaskManager.Task::(prepare) Task=`e9e9e3f5-95a0-42bd-b277-c8d624eb415c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1704::DEBUG::2012-08-30 12:53:32,330::task::588::TaskManager.Task::(_updateState) Task=`e9e9e3f5-95a0-42bd-b277-c8d624eb415c`::moving from state preparing -> state finished >Thread-1704::DEBUG::2012-08-30 12:53:32,330::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1704::DEBUG::2012-08-30 12:53:32,330::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1704::DEBUG::2012-08-30 12:53:32,331::task::978::TaskManager.Task::(_decref) Task=`e9e9e3f5-95a0-42bd-b277-c8d624eb415c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:53:33,134::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:35,176::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1705::DEBUG::2012-08-30 12:53:35,506::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1705::DEBUG::2012-08-30 12:53:35,507::task::588::TaskManager.Task::(_updateState) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::moving from state init -> state preparing >Thread-1705::INFO::2012-08-30 12:53:35,508::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1705::DEBUG::2012-08-30 12:53:35,508::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c5fe1763-638c-4320-ab1b-2a1b4d9a3237`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1705::DEBUG::2012-08-30 12:53:35,508::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1705::DEBUG::2012-08-30 12:53:35,508::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1705::DEBUG::2012-08-30 12:53:35,509::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c5fe1763-638c-4320-ab1b-2a1b4d9a3237`::Granted request >Thread-1705::DEBUG::2012-08-30 12:53:35,509::task::817::TaskManager.Task::(resourceAcquired) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1705::DEBUG::2012-08-30 12:53:35,509::task::978::TaskManager.Task::(_decref) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::ref 1 aborting False >Thread-1705::INFO::2012-08-30 12:53:35,512::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1705::DEBUG::2012-08-30 12:53:35,512::task::1172::TaskManager.Task::(prepare) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1705::DEBUG::2012-08-30 12:53:35,513::task::588::TaskManager.Task::(_updateState) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::moving from state preparing -> state finished >Thread-1705::DEBUG::2012-08-30 12:53:35,513::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1705::DEBUG::2012-08-30 12:53:35,513::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1705::DEBUG::2012-08-30 12:53:35,513::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1705::DEBUG::2012-08-30 12:53:35,514::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1705::DEBUG::2012-08-30 12:53:35,514::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1705::DEBUG::2012-08-30 12:53:35,514::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1705::DEBUG::2012-08-30 12:53:35,514::task::978::TaskManager.Task::(_decref) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:53:38,136::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:38,136::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:40,177::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:43,137::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:45,178::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:45,178::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:48,139::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1709::DEBUG::2012-08-30 12:53:48,855::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1709::DEBUG::2012-08-30 12:53:48,856::task::588::TaskManager.Task::(_updateState) Task=`97dbf53b-308e-4cd3-b6bf-fbd3fd9e04e8`::moving from state init -> state preparing >Thread-1709::INFO::2012-08-30 12:53:48,856::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1709::INFO::2012-08-30 12:53:48,857::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1709::DEBUG::2012-08-30 12:53:48,857::task::1172::TaskManager.Task::(prepare) Task=`97dbf53b-308e-4cd3-b6bf-fbd3fd9e04e8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1709::DEBUG::2012-08-30 12:53:48,857::task::588::TaskManager.Task::(_updateState) Task=`97dbf53b-308e-4cd3-b6bf-fbd3fd9e04e8`::moving from state preparing -> state finished >Thread-1709::DEBUG::2012-08-30 12:53:48,857::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1709::DEBUG::2012-08-30 12:53:48,857::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1709::DEBUG::2012-08-30 12:53:48,858::task::978::TaskManager.Task::(_decref) Task=`97dbf53b-308e-4cd3-b6bf-fbd3fd9e04e8`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:53:50,179::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1711::DEBUG::2012-08-30 12:53:52,058::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1711::DEBUG::2012-08-30 12:53:52,059::task::588::TaskManager.Task::(_updateState) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::moving from state init -> state preparing >Thread-1711::INFO::2012-08-30 12:53:52,059::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1711::DEBUG::2012-08-30 12:53:52,059::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`48b44b7d-9a35-41cf-b21f-96f160f6a8de`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1711::DEBUG::2012-08-30 12:53:52,059::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1711::DEBUG::2012-08-30 12:53:52,060::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1711::DEBUG::2012-08-30 12:53:52,060::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`48b44b7d-9a35-41cf-b21f-96f160f6a8de`::Granted request >Thread-1711::DEBUG::2012-08-30 12:53:52,060::task::817::TaskManager.Task::(resourceAcquired) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1711::DEBUG::2012-08-30 12:53:52,061::task::978::TaskManager.Task::(_decref) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::ref 1 aborting False >Thread-1711::INFO::2012-08-30 12:53:52,063::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1711::DEBUG::2012-08-30 12:53:52,064::task::1172::TaskManager.Task::(prepare) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1711::DEBUG::2012-08-30 12:53:52,064::task::588::TaskManager.Task::(_updateState) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::moving from state preparing -> state finished >Thread-1711::DEBUG::2012-08-30 12:53:52,064::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1711::DEBUG::2012-08-30 12:53:52,064::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1711::DEBUG::2012-08-30 12:53:52,065::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1711::DEBUG::2012-08-30 12:53:52,065::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1711::DEBUG::2012-08-30 12:53:52,065::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1711::DEBUG::2012-08-30 12:53:52,065::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1711::DEBUG::2012-08-30 12:53:52,066::task::978::TaskManager.Task::(_decref) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 12:53:53,140::task::588::TaskManager.Task::(_updateState) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:53:53,140::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:53:53,141::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`df8a4022-932a-43be-ad6d-35cb7779e7be`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:53:53,141::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:53:53,141::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:53:53,141::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`df8a4022-932a-43be-ad6d-35cb7779e7be`::Granted request >Thread-40::DEBUG::2012-08-30 12:53:53,142::task::817::TaskManager.Task::(resourceAcquired) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:53:53,142::task::978::TaskManager.Task::(_decref) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:53:53,144::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:53:53,146::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:53:53,147::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:53:53,147::task::1172::TaskManager.Task::(prepare) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:53:53,147::task::588::TaskManager.Task::(_updateState) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:53:53,148::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:53:53,148::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:53:53,148::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:53:53,148::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:53:53,149::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:53:53,149::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:53:53,149::task::978::TaskManager.Task::(_decref) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:53:53,149::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:53,150::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:53,150::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:53:53,151::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:53:55,180::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:53:55,216::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1712::DEBUG::2012-08-30 12:53:56,487::task::588::TaskManager.Task::(_updateState) Task=`5394ee6e-c1b9-4063-9a53-b201be6aec77`::moving from state init -> state preparing >Thread-1712::INFO::2012-08-30 12:53:56,487::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1712::INFO::2012-08-30 12:53:56,487::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177502632141', 'lastCheck': 1346302429.7076781, 'code': 0, 'valid': True}} >Thread-1712::DEBUG::2012-08-30 12:53:56,488::task::1172::TaskManager.Task::(prepare) Task=`5394ee6e-c1b9-4063-9a53-b201be6aec77`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177502632141', 'lastCheck': 1346302429.7076781, 'code': 0, 'valid': True}} >Thread-1712::DEBUG::2012-08-30 12:53:56,488::task::588::TaskManager.Task::(_updateState) Task=`5394ee6e-c1b9-4063-9a53-b201be6aec77`::moving from state preparing -> state finished >Thread-1712::DEBUG::2012-08-30 12:53:56,488::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1712::DEBUG::2012-08-30 12:53:56,488::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1712::DEBUG::2012-08-30 12:53:56,488::task::978::TaskManager.Task::(_decref) Task=`5394ee6e-c1b9-4063-9a53-b201be6aec77`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:53:58,152::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1713::DEBUG::2012-08-30 12:53:59,693::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1713::DEBUG::2012-08-30 12:53:59,694::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:54:00,181::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:00,182::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:54:02,222::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:54:03,153::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1715::DEBUG::2012-08-30 12:54:04,965::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1715::DEBUG::2012-08-30 12:54:04,966::task::588::TaskManager.Task::(_updateState) Task=`961d92c0-d141-41a2-9e66-3c85efe06717`::moving from state init -> state preparing >Thread-1715::INFO::2012-08-30 12:54:04,966::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1715::INFO::2012-08-30 12:54:04,967::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1715::DEBUG::2012-08-30 12:54:04,967::task::1172::TaskManager.Task::(prepare) Task=`961d92c0-d141-41a2-9e66-3c85efe06717`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1715::DEBUG::2012-08-30 12:54:04,967::task::588::TaskManager.Task::(_updateState) Task=`961d92c0-d141-41a2-9e66-3c85efe06717`::moving from state preparing -> state finished >Thread-1715::DEBUG::2012-08-30 12:54:04,967::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1715::DEBUG::2012-08-30 12:54:04,967::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1715::DEBUG::2012-08-30 12:54:04,968::task::978::TaskManager.Task::(_decref) Task=`961d92c0-d141-41a2-9e66-3c85efe06717`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:54:05,183::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1716::DEBUG::2012-08-30 12:54:08,151::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1716::DEBUG::2012-08-30 12:54:08,151::task::588::TaskManager.Task::(_updateState) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::moving from state init -> state preparing >Thread-1716::INFO::2012-08-30 12:54:08,152::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1716::DEBUG::2012-08-30 12:54:08,152::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`eb532913-abf1-4ffe-9ed7-009dabdb47ae`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1716::DEBUG::2012-08-30 12:54:08,152::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1716::DEBUG::2012-08-30 12:54:08,153::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1716::DEBUG::2012-08-30 12:54:08,153::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`eb532913-abf1-4ffe-9ed7-009dabdb47ae`::Granted request >Thread-1716::DEBUG::2012-08-30 12:54:08,153::task::817::TaskManager.Task::(resourceAcquired) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1716::DEBUG::2012-08-30 12:54:08,154::task::978::TaskManager.Task::(_decref) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::ref 1 aborting False >Thread-40::ERROR::2012-08-30 12:54:08,154::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:08,156::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1716::INFO::2012-08-30 12:54:08,158::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1716::DEBUG::2012-08-30 12:54:08,158::task::1172::TaskManager.Task::(prepare) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1716::DEBUG::2012-08-30 12:54:08,158::task::588::TaskManager.Task::(_updateState) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::moving from state preparing -> state finished >Thread-1716::DEBUG::2012-08-30 12:54:08,158::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1716::DEBUG::2012-08-30 12:54:08,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1716::DEBUG::2012-08-30 12:54:08,159::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1716::DEBUG::2012-08-30 12:54:08,159::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1716::DEBUG::2012-08-30 12:54:08,159::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1716::DEBUG::2012-08-30 12:54:08,160::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1716::DEBUG::2012-08-30 12:54:08,160::task::978::TaskManager.Task::(_decref) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:54:10,183::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:13,158::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:15,184::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:15,185::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:18,160::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:20,186::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1720::DEBUG::2012-08-30 12:54:21,485::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1720::DEBUG::2012-08-30 12:54:21,485::task::588::TaskManager.Task::(_updateState) Task=`60ec2e70-0b9b-4404-b09d-b16636716b9a`::moving from state init -> state preparing >Thread-1720::INFO::2012-08-30 12:54:21,486::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1720::INFO::2012-08-30 12:54:21,486::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1720::DEBUG::2012-08-30 12:54:21,486::task::1172::TaskManager.Task::(prepare) Task=`60ec2e70-0b9b-4404-b09d-b16636716b9a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1720::DEBUG::2012-08-30 12:54:21,486::task::588::TaskManager.Task::(_updateState) Task=`60ec2e70-0b9b-4404-b09d-b16636716b9a`::moving from state preparing -> state finished >Thread-1720::DEBUG::2012-08-30 12:54:21,487::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1720::DEBUG::2012-08-30 12:54:21,487::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1720::DEBUG::2012-08-30 12:54:21,487::task::978::TaskManager.Task::(_decref) Task=`60ec2e70-0b9b-4404-b09d-b16636716b9a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:54:23,161::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:23,162::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1721::DEBUG::2012-08-30 12:54:24,667::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1722::DEBUG::2012-08-30 12:54:24,667::task::588::TaskManager.Task::(_updateState) Task=`aed629d9-9df2-495c-83e7-664aeb51a2a2`::moving from state init -> state preparing >Thread-1721::DEBUG::2012-08-30 12:54:24,668::task::588::TaskManager.Task::(_updateState) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::moving from state init -> state preparing >Thread-1722::INFO::2012-08-30 12:54:24,668::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1721::INFO::2012-08-30 12:54:24,668::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1722::INFO::2012-08-30 12:54:24,668::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346302459.7277031, 'code': 0, 'valid': True}} >Thread-1721::DEBUG::2012-08-30 12:54:24,669::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`11742f64-567e-46cb-8295-6e1e97a0fe9b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1722::DEBUG::2012-08-30 12:54:24,669::task::1172::TaskManager.Task::(prepare) Task=`aed629d9-9df2-495c-83e7-664aeb51a2a2`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346302459.7277031, 'code': 0, 'valid': True}} >Thread-1721::DEBUG::2012-08-30 12:54:24,669::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1722::DEBUG::2012-08-30 12:54:24,669::task::588::TaskManager.Task::(_updateState) Task=`aed629d9-9df2-495c-83e7-664aeb51a2a2`::moving from state preparing -> state finished >Thread-1721::DEBUG::2012-08-30 12:54:24,669::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1722::DEBUG::2012-08-30 12:54:24,670::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1721::DEBUG::2012-08-30 12:54:24,670::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`11742f64-567e-46cb-8295-6e1e97a0fe9b`::Granted request >Thread-1722::DEBUG::2012-08-30 12:54:24,670::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1721::DEBUG::2012-08-30 12:54:24,670::task::817::TaskManager.Task::(resourceAcquired) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1722::DEBUG::2012-08-30 12:54:24,671::task::978::TaskManager.Task::(_decref) Task=`aed629d9-9df2-495c-83e7-664aeb51a2a2`::ref 0 aborting False >Thread-1721::DEBUG::2012-08-30 12:54:24,671::task::978::TaskManager.Task::(_decref) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::ref 1 aborting False >Thread-1721::INFO::2012-08-30 12:54:24,676::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1721::DEBUG::2012-08-30 12:54:24,676::task::1172::TaskManager.Task::(prepare) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1721::DEBUG::2012-08-30 12:54:24,676::task::588::TaskManager.Task::(_updateState) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::moving from state preparing -> state finished >Thread-1721::DEBUG::2012-08-30 12:54:24,676::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1721::DEBUG::2012-08-30 12:54:24,676::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1721::DEBUG::2012-08-30 12:54:24,677::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1721::DEBUG::2012-08-30 12:54:24,677::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1721::DEBUG::2012-08-30 12:54:24,677::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1721::DEBUG::2012-08-30 12:54:24,677::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1721::DEBUG::2012-08-30 12:54:24,678::task::978::TaskManager.Task::(_decref) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:54:25,187::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:54:25,239::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1723::DEBUG::2012-08-30 12:54:27,863::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1723::DEBUG::2012-08-30 12:54:27,863::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 12:54:28,163::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:54:30,188::task::588::TaskManager.Task::(_updateState) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:54:30,188::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:54:30,188::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`fb80bf79-47d0-497b-a2c7-6926f802d73f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:54:30,189::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:54:30,189::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:54:30,189::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`fb80bf79-47d0-497b-a2c7-6926f802d73f`::Granted request >Thread-104::DEBUG::2012-08-30 12:54:30,190::task::817::TaskManager.Task::(resourceAcquired) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:54:30,190::task::978::TaskManager.Task::(_decref) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:54:30,191::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:54:30,193::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:54:30,195::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:54:30,195::task::1172::TaskManager.Task::(prepare) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:54:30,195::task::588::TaskManager.Task::(_updateState) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:54:30,195::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:54:30,195::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:54:30,196::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:54:30,196::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:54:30,196::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:54:30,196::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:54:30,197::task::978::TaskManager.Task::(_decref) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:54:30,197::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:30,198::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:30,198::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:30,198::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:54:32,246::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:54:33,164::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:35,199::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1725::DEBUG::2012-08-30 12:54:37,891::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1725::DEBUG::2012-08-30 12:54:37,892::task::588::TaskManager.Task::(_updateState) Task=`46e09f0e-9d6e-4cc1-ab8f-fe41f20f88d1`::moving from state init -> state preparing >Thread-1725::INFO::2012-08-30 12:54:37,892::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1725::INFO::2012-08-30 12:54:37,892::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1725::DEBUG::2012-08-30 12:54:37,892::task::1172::TaskManager.Task::(prepare) Task=`46e09f0e-9d6e-4cc1-ab8f-fe41f20f88d1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1725::DEBUG::2012-08-30 12:54:37,893::task::588::TaskManager.Task::(_updateState) Task=`46e09f0e-9d6e-4cc1-ab8f-fe41f20f88d1`::moving from state preparing -> state finished >Thread-1725::DEBUG::2012-08-30 12:54:37,893::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1725::DEBUG::2012-08-30 12:54:37,893::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1725::DEBUG::2012-08-30 12:54:37,893::task::978::TaskManager.Task::(_decref) Task=`46e09f0e-9d6e-4cc1-ab8f-fe41f20f88d1`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:54:38,165::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:38,166::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:40,200::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1727::DEBUG::2012-08-30 12:54:41,086::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1727::DEBUG::2012-08-30 12:54:41,086::task::588::TaskManager.Task::(_updateState) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::moving from state init -> state preparing >Thread-1727::INFO::2012-08-30 12:54:41,087::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1727::DEBUG::2012-08-30 12:54:41,087::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ec3d6554-3d37-49f4-8f2e-de4828b8a3bf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1727::DEBUG::2012-08-30 12:54:41,087::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1727::DEBUG::2012-08-30 12:54:41,088::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1727::DEBUG::2012-08-30 12:54:41,088::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ec3d6554-3d37-49f4-8f2e-de4828b8a3bf`::Granted request >Thread-1727::DEBUG::2012-08-30 12:54:41,088::task::817::TaskManager.Task::(resourceAcquired) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1727::DEBUG::2012-08-30 12:54:41,088::task::978::TaskManager.Task::(_decref) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::ref 1 aborting False >Thread-1727::INFO::2012-08-30 12:54:41,091::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1727::DEBUG::2012-08-30 12:54:41,092::task::1172::TaskManager.Task::(prepare) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1727::DEBUG::2012-08-30 12:54:41,092::task::588::TaskManager.Task::(_updateState) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::moving from state preparing -> state finished >Thread-1727::DEBUG::2012-08-30 12:54:41,092::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1727::DEBUG::2012-08-30 12:54:41,092::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1727::DEBUG::2012-08-30 12:54:41,093::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1727::DEBUG::2012-08-30 12:54:41,093::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1727::DEBUG::2012-08-30 12:54:41,093::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1727::DEBUG::2012-08-30 12:54:41,093::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1727::DEBUG::2012-08-30 12:54:41,093::task::978::TaskManager.Task::(_decref) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:54:43,167::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:45,201::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:45,202::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:48,168::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:54:50,203::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 12:54:53,169::task::588::TaskManager.Task::(_updateState) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:54:53,169::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:54:53,170::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4608a04f-68cc-4a87-b7ee-f46adc27d724`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:54:53,170::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:54:53,171::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:54:53,171::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4608a04f-68cc-4a87-b7ee-f46adc27d724`::Granted request >Thread-40::DEBUG::2012-08-30 12:54:53,171::task::817::TaskManager.Task::(resourceAcquired) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:54:53,171::task::978::TaskManager.Task::(_decref) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:54:53,173::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:54:53,175::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:54:53,176::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:54:53,177::task::1172::TaskManager.Task::(prepare) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:54:53,177::task::588::TaskManager.Task::(_updateState) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:54:53,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:54:53,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:54:53,178::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:54:53,178::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:54:53,178::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:54:53,178::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:54:53,179::task::978::TaskManager.Task::(_decref) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:54:53,179::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:53,179::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:53,180::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:54:53,180::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1730::DEBUG::2012-08-30 12:54:54,529::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1730::DEBUG::2012-08-30 12:54:54,529::task::588::TaskManager.Task::(_updateState) Task=`938a4193-075c-4dcf-bef3-f168c8d278eb`::moving from state init -> state preparing >Thread-1730::INFO::2012-08-30 12:54:54,530::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1730::INFO::2012-08-30 12:54:54,530::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1730::DEBUG::2012-08-30 12:54:54,530::task::1172::TaskManager.Task::(prepare) Task=`938a4193-075c-4dcf-bef3-f168c8d278eb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1730::DEBUG::2012-08-30 12:54:54,530::task::588::TaskManager.Task::(_updateState) Task=`938a4193-075c-4dcf-bef3-f168c8d278eb`::moving from state preparing -> state finished >Thread-1730::DEBUG::2012-08-30 12:54:54,531::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1730::DEBUG::2012-08-30 12:54:54,531::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1730::DEBUG::2012-08-30 12:54:54,531::task::978::TaskManager.Task::(_decref) Task=`938a4193-075c-4dcf-bef3-f168c8d278eb`::ref 0 aborting False >Thread-1731::DEBUG::2012-08-30 12:54:54,532::task::588::TaskManager.Task::(_updateState) Task=`1217bca7-b05c-413d-a105-909846ee2064`::moving from state init -> state preparing >Thread-1731::INFO::2012-08-30 12:54:54,532::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1731::INFO::2012-08-30 12:54:54,533::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00196695327759', 'lastCheck': 1346302489.7485011, 'code': 0, 'valid': True}} >Thread-1731::DEBUG::2012-08-30 12:54:54,533::task::1172::TaskManager.Task::(prepare) Task=`1217bca7-b05c-413d-a105-909846ee2064`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00196695327759', 'lastCheck': 1346302489.7485011, 'code': 0, 'valid': True}} >Thread-1731::DEBUG::2012-08-30 12:54:54,533::task::588::TaskManager.Task::(_updateState) Task=`1217bca7-b05c-413d-a105-909846ee2064`::moving from state preparing -> state finished >Thread-1731::DEBUG::2012-08-30 12:54:54,533::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1731::DEBUG::2012-08-30 12:54:54,534::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1731::DEBUG::2012-08-30 12:54:54,534::task::978::TaskManager.Task::(_decref) Task=`1217bca7-b05c-413d-a105-909846ee2064`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:54:55,204::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:54:55,266::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1732::DEBUG::2012-08-30 12:54:57,875::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1732::DEBUG::2012-08-30 12:54:57,876::task::588::TaskManager.Task::(_updateState) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::moving from state init -> state preparing >Thread-1732::INFO::2012-08-30 12:54:57,876::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1732::DEBUG::2012-08-30 12:54:57,876::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`99ca25af-061c-4a07-946e-557f65600d22`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1732::DEBUG::2012-08-30 12:54:57,876::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1732::DEBUG::2012-08-30 12:54:57,877::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1732::DEBUG::2012-08-30 12:54:57,877::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`99ca25af-061c-4a07-946e-557f65600d22`::Granted request >Thread-1732::DEBUG::2012-08-30 12:54:57,877::task::817::TaskManager.Task::(resourceAcquired) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1732::DEBUG::2012-08-30 12:54:57,878::task::978::TaskManager.Task::(_decref) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::ref 1 aborting False >Thread-1733::DEBUG::2012-08-30 12:54:57,879::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1733::DEBUG::2012-08-30 12:54:57,879::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1732::INFO::2012-08-30 12:54:57,883::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1732::DEBUG::2012-08-30 12:54:57,883::task::1172::TaskManager.Task::(prepare) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1732::DEBUG::2012-08-30 12:54:57,883::task::588::TaskManager.Task::(_updateState) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::moving from state preparing -> state finished >Thread-1732::DEBUG::2012-08-30 12:54:57,884::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1732::DEBUG::2012-08-30 12:54:57,884::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1732::DEBUG::2012-08-30 12:54:57,884::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1732::DEBUG::2012-08-30 12:54:57,884::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1732::DEBUG::2012-08-30 12:54:57,885::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1732::DEBUG::2012-08-30 12:54:57,885::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1732::DEBUG::2012-08-30 12:54:57,885::task::978::TaskManager.Task::(_decref) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:54:58,181::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:00,205::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:00,205::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:55:02,274::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:55:03,183::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:05,206::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:08,185::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:08,185::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:10,207::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1736::DEBUG::2012-08-30 12:55:11,131::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1736::DEBUG::2012-08-30 12:55:11,132::task::588::TaskManager.Task::(_updateState) Task=`6ecf885b-8154-42fa-b236-f56e4c36ef99`::moving from state init -> state preparing >Thread-1736::INFO::2012-08-30 12:55:11,132::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1736::INFO::2012-08-30 12:55:11,132::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1736::DEBUG::2012-08-30 12:55:11,132::task::1172::TaskManager.Task::(prepare) Task=`6ecf885b-8154-42fa-b236-f56e4c36ef99`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1736::DEBUG::2012-08-30 12:55:11,133::task::588::TaskManager.Task::(_updateState) Task=`6ecf885b-8154-42fa-b236-f56e4c36ef99`::moving from state preparing -> state finished >Thread-1736::DEBUG::2012-08-30 12:55:11,133::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1736::DEBUG::2012-08-30 12:55:11,133::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1736::DEBUG::2012-08-30 12:55:11,133::task::978::TaskManager.Task::(_decref) Task=`6ecf885b-8154-42fa-b236-f56e4c36ef99`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:55:13,186::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1738::DEBUG::2012-08-30 12:55:14,327::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1738::DEBUG::2012-08-30 12:55:14,328::task::588::TaskManager.Task::(_updateState) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::moving from state init -> state preparing >Thread-1738::INFO::2012-08-30 12:55:14,328::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1738::DEBUG::2012-08-30 12:55:14,328::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ba971fc1-a04b-4c46-a6e0-6d8707060c2e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1738::DEBUG::2012-08-30 12:55:14,328::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1738::DEBUG::2012-08-30 12:55:14,329::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1738::DEBUG::2012-08-30 12:55:14,329::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ba971fc1-a04b-4c46-a6e0-6d8707060c2e`::Granted request >Thread-1738::DEBUG::2012-08-30 12:55:14,329::task::817::TaskManager.Task::(resourceAcquired) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1738::DEBUG::2012-08-30 12:55:14,329::task::978::TaskManager.Task::(_decref) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::ref 1 aborting False >Thread-1738::INFO::2012-08-30 12:55:14,332::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1738::DEBUG::2012-08-30 12:55:14,333::task::1172::TaskManager.Task::(prepare) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1738::DEBUG::2012-08-30 12:55:14,333::task::588::TaskManager.Task::(_updateState) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::moving from state preparing -> state finished >Thread-1738::DEBUG::2012-08-30 12:55:14,333::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1738::DEBUG::2012-08-30 12:55:14,333::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1738::DEBUG::2012-08-30 12:55:14,334::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1738::DEBUG::2012-08-30 12:55:14,334::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1738::DEBUG::2012-08-30 12:55:14,334::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1738::DEBUG::2012-08-30 12:55:14,334::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1738::DEBUG::2012-08-30 12:55:14,335::task::978::TaskManager.Task::(_decref) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:55:15,208::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:15,208::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:18,187::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:20,209::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:23,189::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:23,189::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1740::DEBUG::2012-08-30 12:55:23,950::task::588::TaskManager.Task::(_updateState) Task=`3531e501-5d39-419c-a175-98a5051661eb`::moving from state init -> state preparing >Thread-1740::INFO::2012-08-30 12:55:23,951::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1740::INFO::2012-08-30 12:55:23,951::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016188621521', 'lastCheck': 1346302519.7696941, 'code': 0, 'valid': True}} >Thread-1740::DEBUG::2012-08-30 12:55:23,951::task::1172::TaskManager.Task::(prepare) Task=`3531e501-5d39-419c-a175-98a5051661eb`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016188621521', 'lastCheck': 1346302519.7696941, 'code': 0, 'valid': True}} >Thread-1740::DEBUG::2012-08-30 12:55:23,951::task::588::TaskManager.Task::(_updateState) Task=`3531e501-5d39-419c-a175-98a5051661eb`::moving from state preparing -> state finished >Thread-1740::DEBUG::2012-08-30 12:55:23,952::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1740::DEBUG::2012-08-30 12:55:23,952::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1740::DEBUG::2012-08-30 12:55:23,952::task::978::TaskManager.Task::(_decref) Task=`3531e501-5d39-419c-a175-98a5051661eb`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:55:25,210::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:55:25,295::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1741::DEBUG::2012-08-30 12:55:27,258::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1741::DEBUG::2012-08-30 12:55:27,258::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1742::DEBUG::2012-08-30 12:55:27,260::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1742::DEBUG::2012-08-30 12:55:27,260::task::588::TaskManager.Task::(_updateState) Task=`6fe4c3ab-add0-4f53-9486-74cc41bd373d`::moving from state init -> state preparing >Thread-1742::INFO::2012-08-30 12:55:27,261::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1742::INFO::2012-08-30 12:55:27,261::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1742::DEBUG::2012-08-30 12:55:27,261::task::1172::TaskManager.Task::(prepare) Task=`6fe4c3ab-add0-4f53-9486-74cc41bd373d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1742::DEBUG::2012-08-30 12:55:27,261::task::588::TaskManager.Task::(_updateState) Task=`6fe4c3ab-add0-4f53-9486-74cc41bd373d`::moving from state preparing -> state finished >Thread-1742::DEBUG::2012-08-30 12:55:27,262::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1742::DEBUG::2012-08-30 12:55:27,262::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1742::DEBUG::2012-08-30 12:55:27,262::task::978::TaskManager.Task::(_decref) Task=`6fe4c3ab-add0-4f53-9486-74cc41bd373d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:55:28,191::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-26::DEBUG::2012-08-30 12:55:29,769::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 12:55:29,770::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-26::DEBUG::2012-08-30 12:55:29,770::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None) >Thread-26::DEBUG::2012-08-30 12:55:29,891::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5 >Thread-26::WARNING::2012-08-30 12:55:29,893::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found'] >Thread-26::DEBUG::2012-08-30 12:55:29,893::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-26::DEBUG::2012-08-30 12:55:29,899::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 12:55:29,899::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend >Thread-26::DEBUG::2012-08-30 12:55:29,902::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d'] >Thread-26::DEBUG::2012-08-30 12:55:29,903::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-26::WARNING::2012-08-30 12:55:29,903::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered >Thread-26::WARNING::2012-08-30 12:55:29,903::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered >Thread-104::DEBUG::2012-08-30 12:55:30,211::task::588::TaskManager.Task::(_updateState) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:55:30,212::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:55:30,212::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a53bbf85-da13-4ccd-b431-e7a28ff5c523`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:55:30,212::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:55:30,213::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:55:30,213::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a53bbf85-da13-4ccd-b431-e7a28ff5c523`::Granted request >Thread-104::DEBUG::2012-08-30 12:55:30,213::task::817::TaskManager.Task::(resourceAcquired) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:55:30,213::task::978::TaskManager.Task::(_decref) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:55:30,215::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:55:30,217::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:55:30,218::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:55:30,219::task::1172::TaskManager.Task::(prepare) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:55:30,219::task::588::TaskManager.Task::(_updateState) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:55:30,219::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:55:30,219::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:55:30,220::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:55:30,220::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:55:30,220::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:55:30,220::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:55:30,220::task::978::TaskManager.Task::(_decref) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:55:30,221::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:30,221::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:30,222::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:30,222::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1743::DEBUG::2012-08-30 12:55:30,455::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1743::DEBUG::2012-08-30 12:55:30,456::task::588::TaskManager.Task::(_updateState) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::moving from state init -> state preparing >Thread-1743::INFO::2012-08-30 12:55:30,456::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1743::DEBUG::2012-08-30 12:55:30,456::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d971836a-854e-422d-8db5-efe4fd9bb2f4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1743::DEBUG::2012-08-30 12:55:30,457::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1743::DEBUG::2012-08-30 12:55:30,457::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1743::DEBUG::2012-08-30 12:55:30,457::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d971836a-854e-422d-8db5-efe4fd9bb2f4`::Granted request >Thread-1743::DEBUG::2012-08-30 12:55:30,457::task::817::TaskManager.Task::(resourceAcquired) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1743::DEBUG::2012-08-30 12:55:30,458::task::978::TaskManager.Task::(_decref) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::ref 1 aborting False >Thread-1743::INFO::2012-08-30 12:55:30,461::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1743::DEBUG::2012-08-30 12:55:30,461::task::1172::TaskManager.Task::(prepare) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1743::DEBUG::2012-08-30 12:55:30,461::task::588::TaskManager.Task::(_updateState) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::moving from state preparing -> state finished >Thread-1743::DEBUG::2012-08-30 12:55:30,461::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1743::DEBUG::2012-08-30 12:55:30,463::task::978::TaskManager.Task::(_decref) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:55:32,302::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:55:33,192::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:35,223::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:38,193::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:38,193::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:40,224::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:43,194::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1752::DEBUG::2012-08-30 12:55:43,686::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1752::DEBUG::2012-08-30 12:55:43,686::task::588::TaskManager.Task::(_updateState) Task=`88738fb9-06f6-4fef-9d6f-40b1b82994e2`::moving from state init -> state preparing >Thread-1752::INFO::2012-08-30 12:55:43,687::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1752::INFO::2012-08-30 12:55:43,687::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1752::DEBUG::2012-08-30 12:55:43,687::task::1172::TaskManager.Task::(prepare) Task=`88738fb9-06f6-4fef-9d6f-40b1b82994e2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1752::DEBUG::2012-08-30 12:55:43,687::task::588::TaskManager.Task::(_updateState) Task=`88738fb9-06f6-4fef-9d6f-40b1b82994e2`::moving from state preparing -> state finished >Thread-1752::DEBUG::2012-08-30 12:55:43,688::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1752::DEBUG::2012-08-30 12:55:43,688::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1752::DEBUG::2012-08-30 12:55:43,688::task::978::TaskManager.Task::(_decref) Task=`88738fb9-06f6-4fef-9d6f-40b1b82994e2`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:55:45,226::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:45,226::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1753::DEBUG::2012-08-30 12:55:46,883::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1753::DEBUG::2012-08-30 12:55:46,884::task::588::TaskManager.Task::(_updateState) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::moving from state init -> state preparing >Thread-1753::INFO::2012-08-30 12:55:46,884::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1753::DEBUG::2012-08-30 12:55:46,885::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`bfe92591-7d04-400e-8ca5-98ed1c09ae1c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1753::DEBUG::2012-08-30 12:55:46,885::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1753::DEBUG::2012-08-30 12:55:46,885::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1753::DEBUG::2012-08-30 12:55:46,886::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`bfe92591-7d04-400e-8ca5-98ed1c09ae1c`::Granted request >Thread-1753::DEBUG::2012-08-30 12:55:46,886::task::817::TaskManager.Task::(resourceAcquired) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1753::DEBUG::2012-08-30 12:55:46,887::task::978::TaskManager.Task::(_decref) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::ref 1 aborting False >Thread-1753::INFO::2012-08-30 12:55:46,890::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1753::DEBUG::2012-08-30 12:55:46,890::task::1172::TaskManager.Task::(prepare) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1753::DEBUG::2012-08-30 12:55:46,890::task::588::TaskManager.Task::(_updateState) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::moving from state preparing -> state finished >Thread-1753::DEBUG::2012-08-30 12:55:46,890::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1753::DEBUG::2012-08-30 12:55:46,891::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1753::DEBUG::2012-08-30 12:55:46,891::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1753::DEBUG::2012-08-30 12:55:46,891::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1753::DEBUG::2012-08-30 12:55:46,891::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1753::DEBUG::2012-08-30 12:55:46,892::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1753::DEBUG::2012-08-30 12:55:46,892::task::978::TaskManager.Task::(_decref) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:55:48,195::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:50,227::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1755::DEBUG::2012-08-30 12:55:52,067::task::588::TaskManager.Task::(_updateState) Task=`b6131399-df4c-461b-bfc7-31098b94103f`::moving from state init -> state preparing >Thread-1755::INFO::2012-08-30 12:55:52,067::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1755::INFO::2012-08-30 12:55:52,068::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00175499916077', 'lastCheck': 1346302549.9230909, 'code': 0, 'valid': True}} >Thread-1755::DEBUG::2012-08-30 12:55:52,068::task::1172::TaskManager.Task::(prepare) Task=`b6131399-df4c-461b-bfc7-31098b94103f`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00175499916077', 'lastCheck': 1346302549.9230909, 'code': 0, 'valid': True}} >Thread-1755::DEBUG::2012-08-30 12:55:52,068::task::588::TaskManager.Task::(_updateState) Task=`b6131399-df4c-461b-bfc7-31098b94103f`::moving from state preparing -> state finished >Thread-1755::DEBUG::2012-08-30 12:55:52,068::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1755::DEBUG::2012-08-30 12:55:52,068::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1755::DEBUG::2012-08-30 12:55:52,069::task::978::TaskManager.Task::(_decref) Task=`b6131399-df4c-461b-bfc7-31098b94103f`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 12:55:53,196::task::588::TaskManager.Task::(_updateState) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:55:53,197::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:55:53,197::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9c1c1f73-e56d-41fe-b7f9-36096cf3fa2d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:55:53,197::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:55:53,197::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:55:53,198::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9c1c1f73-e56d-41fe-b7f9-36096cf3fa2d`::Granted request >Thread-40::DEBUG::2012-08-30 12:55:53,198::task::817::TaskManager.Task::(resourceAcquired) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:55:53,198::task::978::TaskManager.Task::(_decref) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:55:53,200::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:55:53,202::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:55:53,203::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:55:53,203::task::1172::TaskManager.Task::(prepare) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:55:53,204::task::588::TaskManager.Task::(_updateState) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:55:53,204::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:55:53,204::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:55:53,204::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:55:53,205::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:55:53,205::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:55:53,205::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:55:53,205::task::978::TaskManager.Task::(_decref) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:55:53,206::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:53,206::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:53,207::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:55:53,207::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:55:55,229::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1756::DEBUG::2012-08-30 12:55:55,278::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1756::DEBUG::2012-08-30 12:55:55,278::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >VM Channels Listener::DEBUG::2012-08-30 12:55:55,322::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 12:55:58,208::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:00,230::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:00,231::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1757::DEBUG::2012-08-30 12:56:00,358::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1757::DEBUG::2012-08-30 12:56:00,359::task::588::TaskManager.Task::(_updateState) Task=`b20025d5-6ccd-4f15-baec-a38e287d30d4`::moving from state init -> state preparing >Thread-1757::INFO::2012-08-30 12:56:00,359::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1757::INFO::2012-08-30 12:56:00,359::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1757::DEBUG::2012-08-30 12:56:00,360::task::1172::TaskManager.Task::(prepare) Task=`b20025d5-6ccd-4f15-baec-a38e287d30d4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1757::DEBUG::2012-08-30 12:56:00,360::task::588::TaskManager.Task::(_updateState) Task=`b20025d5-6ccd-4f15-baec-a38e287d30d4`::moving from state preparing -> state finished >Thread-1757::DEBUG::2012-08-30 12:56:00,360::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1757::DEBUG::2012-08-30 12:56:00,361::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1757::DEBUG::2012-08-30 12:56:00,361::task::978::TaskManager.Task::(_decref) Task=`b20025d5-6ccd-4f15-baec-a38e287d30d4`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 12:56:02,329::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:56:03,209::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1759::DEBUG::2012-08-30 12:56:03,562::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1759::DEBUG::2012-08-30 12:56:03,563::task::588::TaskManager.Task::(_updateState) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::moving from state init -> state preparing >Thread-1759::INFO::2012-08-30 12:56:03,563::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1759::DEBUG::2012-08-30 12:56:03,563::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`86f06304-3a57-44bb-ae4c-cf6961fe460a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1759::DEBUG::2012-08-30 12:56:03,564::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1759::DEBUG::2012-08-30 12:56:03,564::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1759::DEBUG::2012-08-30 12:56:03,564::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`86f06304-3a57-44bb-ae4c-cf6961fe460a`::Granted request >Thread-1759::DEBUG::2012-08-30 12:56:03,565::task::817::TaskManager.Task::(resourceAcquired) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1759::DEBUG::2012-08-30 12:56:03,565::task::978::TaskManager.Task::(_decref) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::ref 1 aborting False >Thread-1759::INFO::2012-08-30 12:56:03,568::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1759::DEBUG::2012-08-30 12:56:03,568::task::1172::TaskManager.Task::(prepare) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1759::DEBUG::2012-08-30 12:56:03,568::task::588::TaskManager.Task::(_updateState) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::moving from state preparing -> state finished >Thread-1759::DEBUG::2012-08-30 12:56:03,568::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1759::DEBUG::2012-08-30 12:56:03,569::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1759::DEBUG::2012-08-30 12:56:03,569::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1759::DEBUG::2012-08-30 12:56:03,569::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1759::DEBUG::2012-08-30 12:56:03,569::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1759::DEBUG::2012-08-30 12:56:03,570::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1759::DEBUG::2012-08-30 12:56:03,570::task::978::TaskManager.Task::(_decref) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:56:05,232::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:08,210::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:08,210::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:10,233::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:13,211::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:15,234::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:15,235::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1763::DEBUG::2012-08-30 12:56:16,858::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1763::DEBUG::2012-08-30 12:56:16,859::task::588::TaskManager.Task::(_updateState) Task=`7301efec-f069-4ccc-b293-77766d2e32bd`::moving from state init -> state preparing >Thread-1763::INFO::2012-08-30 12:56:16,859::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1763::INFO::2012-08-30 12:56:16,859::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1763::DEBUG::2012-08-30 12:56:16,860::task::1172::TaskManager.Task::(prepare) Task=`7301efec-f069-4ccc-b293-77766d2e32bd`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1763::DEBUG::2012-08-30 12:56:16,860::task::588::TaskManager.Task::(_updateState) Task=`7301efec-f069-4ccc-b293-77766d2e32bd`::moving from state preparing -> state finished >Thread-1763::DEBUG::2012-08-30 12:56:16,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1763::DEBUG::2012-08-30 12:56:16,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1763::DEBUG::2012-08-30 12:56:16,861::task::978::TaskManager.Task::(_decref) Task=`7301efec-f069-4ccc-b293-77766d2e32bd`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:56:18,212::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1764::DEBUG::2012-08-30 12:56:20,060::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1765::DEBUG::2012-08-30 12:56:20,060::task::588::TaskManager.Task::(_updateState) Task=`68c756e7-b4f5-427d-87c3-2f3058216350`::moving from state init -> state preparing >Thread-1764::DEBUG::2012-08-30 12:56:20,060::task::588::TaskManager.Task::(_updateState) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::moving from state init -> state preparing >Thread-1765::INFO::2012-08-30 12:56:20,060::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1764::INFO::2012-08-30 12:56:20,061::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1765::INFO::2012-08-30 12:56:20,061::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00160503387451', 'lastCheck': 1346302579.9429951, 'code': 0, 'valid': True}} >Thread-1764::DEBUG::2012-08-30 12:56:20,061::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`23a0ff0f-a409-4894-be5e-260639ef9ca7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1765::DEBUG::2012-08-30 12:56:20,061::task::1172::TaskManager.Task::(prepare) Task=`68c756e7-b4f5-427d-87c3-2f3058216350`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00160503387451', 'lastCheck': 1346302579.9429951, 'code': 0, 'valid': True}} >Thread-1764::DEBUG::2012-08-30 12:56:20,062::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1765::DEBUG::2012-08-30 12:56:20,062::task::588::TaskManager.Task::(_updateState) Task=`68c756e7-b4f5-427d-87c3-2f3058216350`::moving from state preparing -> state finished >Thread-1764::DEBUG::2012-08-30 12:56:20,062::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1765::DEBUG::2012-08-30 12:56:20,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1764::DEBUG::2012-08-30 12:56:20,062::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`23a0ff0f-a409-4894-be5e-260639ef9ca7`::Granted request >Thread-1765::DEBUG::2012-08-30 12:56:20,063::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1764::DEBUG::2012-08-30 12:56:20,063::task::817::TaskManager.Task::(resourceAcquired) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1765::DEBUG::2012-08-30 12:56:20,063::task::978::TaskManager.Task::(_decref) Task=`68c756e7-b4f5-427d-87c3-2f3058216350`::ref 0 aborting False >Thread-1764::DEBUG::2012-08-30 12:56:20,063::task::978::TaskManager.Task::(_decref) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::ref 1 aborting False >Thread-1764::INFO::2012-08-30 12:56:20,068::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1764::DEBUG::2012-08-30 12:56:20,068::task::1172::TaskManager.Task::(prepare) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1764::DEBUG::2012-08-30 12:56:20,069::task::588::TaskManager.Task::(_updateState) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::moving from state preparing -> state finished >Thread-1764::DEBUG::2012-08-30 12:56:20,069::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1764::DEBUG::2012-08-30 12:56:20,069::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1764::DEBUG::2012-08-30 12:56:20,070::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1764::DEBUG::2012-08-30 12:56:20,070::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1764::DEBUG::2012-08-30 12:56:20,070::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1764::DEBUG::2012-08-30 12:56:20,070::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1764::DEBUG::2012-08-30 12:56:20,070::task::978::TaskManager.Task::(_decref) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:56:20,235::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:23,213::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:23,214::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1766::DEBUG::2012-08-30 12:56:23,523::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1766::DEBUG::2012-08-30 12:56:23,523::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:56:25,237::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:56:25,346::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 12:56:28,214::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:56:30,238::task::588::TaskManager.Task::(_updateState) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:56:30,238::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:56:30,238::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e39dceab-9c0e-43e6-9abc-cfe3900702b0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:56:30,239::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:56:30,239::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:56:30,239::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e39dceab-9c0e-43e6-9abc-cfe3900702b0`::Granted request >Thread-104::DEBUG::2012-08-30 12:56:30,239::task::817::TaskManager.Task::(resourceAcquired) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:56:30,240::task::978::TaskManager.Task::(_decref) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:56:30,241::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:56:30,243::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:56:30,244::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:56:30,245::task::1172::TaskManager.Task::(prepare) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:56:30,245::task::588::TaskManager.Task::(_updateState) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:56:30,245::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:56:30,245::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:56:30,246::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:56:30,246::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:56:30,246::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:56:30,246::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:56:30,246::task::978::TaskManager.Task::(_decref) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:56:30,247::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:30,247::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:30,248::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:30,248::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:56:32,348::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:56:33,215::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:35,249::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1768::DEBUG::2012-08-30 12:56:35,293::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1768::DEBUG::2012-08-30 12:56:35,294::task::588::TaskManager.Task::(_updateState) Task=`d7e30343-86b5-4ca2-bb19-ac46babb8ae9`::moving from state init -> state preparing >Thread-1768::INFO::2012-08-30 12:56:35,295::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1768::INFO::2012-08-30 12:56:35,295::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1768::DEBUG::2012-08-30 12:56:35,295::task::1172::TaskManager.Task::(prepare) Task=`d7e30343-86b5-4ca2-bb19-ac46babb8ae9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1768::DEBUG::2012-08-30 12:56:35,296::task::588::TaskManager.Task::(_updateState) Task=`d7e30343-86b5-4ca2-bb19-ac46babb8ae9`::moving from state preparing -> state finished >Thread-1768::DEBUG::2012-08-30 12:56:35,296::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1768::DEBUG::2012-08-30 12:56:35,296::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1768::DEBUG::2012-08-30 12:56:35,296::task::978::TaskManager.Task::(_decref) Task=`d7e30343-86b5-4ca2-bb19-ac46babb8ae9`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:56:38,216::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:38,217::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1770::DEBUG::2012-08-30 12:56:38,493::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1770::DEBUG::2012-08-30 12:56:38,494::task::588::TaskManager.Task::(_updateState) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::moving from state init -> state preparing >Thread-1770::INFO::2012-08-30 12:56:38,494::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1770::DEBUG::2012-08-30 12:56:38,495::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c268c2ef-27b6-49fb-9d58-8edf19e4cd46`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1770::DEBUG::2012-08-30 12:56:38,495::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1770::DEBUG::2012-08-30 12:56:38,495::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1770::DEBUG::2012-08-30 12:56:38,495::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c268c2ef-27b6-49fb-9d58-8edf19e4cd46`::Granted request >Thread-1770::DEBUG::2012-08-30 12:56:38,496::task::817::TaskManager.Task::(resourceAcquired) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1770::DEBUG::2012-08-30 12:56:38,496::task::978::TaskManager.Task::(_decref) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::ref 1 aborting False >Thread-1770::INFO::2012-08-30 12:56:38,499::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1770::DEBUG::2012-08-30 12:56:38,499::task::1172::TaskManager.Task::(prepare) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1770::DEBUG::2012-08-30 12:56:38,499::task::588::TaskManager.Task::(_updateState) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::moving from state preparing -> state finished >Thread-1770::DEBUG::2012-08-30 12:56:38,499::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1770::DEBUG::2012-08-30 12:56:38,500::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1770::DEBUG::2012-08-30 12:56:38,500::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1770::DEBUG::2012-08-30 12:56:38,500::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1770::DEBUG::2012-08-30 12:56:38,500::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1770::DEBUG::2012-08-30 12:56:38,501::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1770::DEBUG::2012-08-30 12:56:38,501::task::978::TaskManager.Task::(_decref) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:56:40,250::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:43,218::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:45,252::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:56:45,252::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:48,219::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1773::DEBUG::2012-08-30 12:56:50,166::task::588::TaskManager.Task::(_updateState) Task=`57a3bbd6-73fb-429e-baf9-83d9fe078880`::moving from state init -> state preparing >Thread-1773::INFO::2012-08-30 12:56:50,166::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1773::INFO::2012-08-30 12:56:50,167::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176787376404', 'lastCheck': 1346302609.962445, 'code': 0, 'valid': True}} >Thread-1773::DEBUG::2012-08-30 12:56:50,167::task::1172::TaskManager.Task::(prepare) Task=`57a3bbd6-73fb-429e-baf9-83d9fe078880`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176787376404', 'lastCheck': 1346302609.962445, 'code': 0, 'valid': True}} >Thread-1773::DEBUG::2012-08-30 12:56:50,167::task::588::TaskManager.Task::(_updateState) Task=`57a3bbd6-73fb-429e-baf9-83d9fe078880`::moving from state preparing -> state finished >Thread-1773::DEBUG::2012-08-30 12:56:50,167::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1773::DEBUG::2012-08-30 12:56:50,168::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1773::DEBUG::2012-08-30 12:56:50,168::task::978::TaskManager.Task::(_decref) Task=`57a3bbd6-73fb-429e-baf9-83d9fe078880`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:56:50,254::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1774::DEBUG::2012-08-30 12:56:51,760::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1774::DEBUG::2012-08-30 12:56:51,761::task::588::TaskManager.Task::(_updateState) Task=`f2756b39-7b44-4b46-ae42-4eed386cc7b0`::moving from state init -> state preparing >Thread-1774::INFO::2012-08-30 12:56:51,761::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1774::INFO::2012-08-30 12:56:51,761::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1774::DEBUG::2012-08-30 12:56:51,762::task::1172::TaskManager.Task::(prepare) Task=`f2756b39-7b44-4b46-ae42-4eed386cc7b0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1774::DEBUG::2012-08-30 12:56:51,762::task::588::TaskManager.Task::(_updateState) Task=`f2756b39-7b44-4b46-ae42-4eed386cc7b0`::moving from state preparing -> state finished >Thread-1774::DEBUG::2012-08-30 12:56:51,762::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1774::DEBUG::2012-08-30 12:56:51,762::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1774::DEBUG::2012-08-30 12:56:51,763::task::978::TaskManager.Task::(_decref) Task=`f2756b39-7b44-4b46-ae42-4eed386cc7b0`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 12:56:53,220::task::588::TaskManager.Task::(_updateState) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:56:53,220::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:56:53,221::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`477cd298-0633-46b9-b114-8d4a4e5a2920`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:56:53,221::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:56:53,221::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:56:53,221::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`477cd298-0633-46b9-b114-8d4a4e5a2920`::Granted request >Thread-40::DEBUG::2012-08-30 12:56:53,222::task::817::TaskManager.Task::(resourceAcquired) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:56:53,222::task::978::TaskManager.Task::(_decref) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:56:53,224::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:56:53,226::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:56:53,227::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:56:53,227::task::1172::TaskManager.Task::(prepare) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:56:53,227::task::588::TaskManager.Task::(_updateState) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:56:53,228::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:56:53,228::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:56:53,228::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:56:53,228::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:56:53,229::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:56:53,229::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:56:53,229::task::978::TaskManager.Task::(_decref) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:56:53,229::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:53,230::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:53,230::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:56:53,231::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1775::DEBUG::2012-08-30 12:56:53,350::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1775::DEBUG::2012-08-30 12:56:53,350::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1776::DEBUG::2012-08-30 12:56:54,940::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1776::DEBUG::2012-08-30 12:56:54,940::task::588::TaskManager.Task::(_updateState) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::moving from state init -> state preparing >Thread-1776::INFO::2012-08-30 12:56:54,940::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1776::DEBUG::2012-08-30 12:56:54,941::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a011c9e7-4d3f-4e45-95aa-2a0059c11a89`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1776::DEBUG::2012-08-30 12:56:54,941::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1776::DEBUG::2012-08-30 12:56:54,941::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1776::DEBUG::2012-08-30 12:56:54,941::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a011c9e7-4d3f-4e45-95aa-2a0059c11a89`::Granted request >Thread-1776::DEBUG::2012-08-30 12:56:54,942::task::817::TaskManager.Task::(resourceAcquired) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1776::DEBUG::2012-08-30 12:56:54,942::task::978::TaskManager.Task::(_decref) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::ref 1 aborting False >Thread-1776::INFO::2012-08-30 12:56:54,945::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1776::DEBUG::2012-08-30 12:56:54,945::task::1172::TaskManager.Task::(prepare) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1776::DEBUG::2012-08-30 12:56:54,945::task::588::TaskManager.Task::(_updateState) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::moving from state preparing -> state finished >Thread-1776::DEBUG::2012-08-30 12:56:54,946::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1776::DEBUG::2012-08-30 12:56:54,946::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1776::DEBUG::2012-08-30 12:56:54,946::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1776::DEBUG::2012-08-30 12:56:54,946::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1776::DEBUG::2012-08-30 12:56:54,947::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1776::DEBUG::2012-08-30 12:56:54,947::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1776::DEBUG::2012-08-30 12:56:54,947::task::978::TaskManager.Task::(_decref) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:56:55,255::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:56:55,369::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 12:56:58,232::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:00,256::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:00,257::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:57:02,375::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:57:03,233::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:05,258::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:08,233::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:08,234::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1779::DEBUG::2012-08-30 12:57:08,239::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1779::DEBUG::2012-08-30 12:57:08,240::task::588::TaskManager.Task::(_updateState) Task=`276c7be3-f971-4b66-848e-c126db0e8a7f`::moving from state init -> state preparing >Thread-1779::INFO::2012-08-30 12:57:08,240::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1779::INFO::2012-08-30 12:57:08,240::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1779::DEBUG::2012-08-30 12:57:08,240::task::1172::TaskManager.Task::(prepare) Task=`276c7be3-f971-4b66-848e-c126db0e8a7f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1779::DEBUG::2012-08-30 12:57:08,241::task::588::TaskManager.Task::(_updateState) Task=`276c7be3-f971-4b66-848e-c126db0e8a7f`::moving from state preparing -> state finished >Thread-1779::DEBUG::2012-08-30 12:57:08,241::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1779::DEBUG::2012-08-30 12:57:08,241::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1779::DEBUG::2012-08-30 12:57:08,241::task::978::TaskManager.Task::(_decref) Task=`276c7be3-f971-4b66-848e-c126db0e8a7f`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:57:10,259::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1781::DEBUG::2012-08-30 12:57:11,429::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1781::DEBUG::2012-08-30 12:57:11,430::task::588::TaskManager.Task::(_updateState) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::moving from state init -> state preparing >Thread-1781::INFO::2012-08-30 12:57:11,430::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1781::DEBUG::2012-08-30 12:57:11,431::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e39b3495-dc36-45cd-91d4-f955d1ba5edb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1781::DEBUG::2012-08-30 12:57:11,431::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1781::DEBUG::2012-08-30 12:57:11,431::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1781::DEBUG::2012-08-30 12:57:11,431::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e39b3495-dc36-45cd-91d4-f955d1ba5edb`::Granted request >Thread-1781::DEBUG::2012-08-30 12:57:11,432::task::817::TaskManager.Task::(resourceAcquired) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1781::DEBUG::2012-08-30 12:57:11,432::task::978::TaskManager.Task::(_decref) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::ref 1 aborting False >Thread-1781::INFO::2012-08-30 12:57:11,435::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1781::DEBUG::2012-08-30 12:57:11,435::task::1172::TaskManager.Task::(prepare) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1781::DEBUG::2012-08-30 12:57:11,435::task::588::TaskManager.Task::(_updateState) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::moving from state preparing -> state finished >Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1781::DEBUG::2012-08-30 12:57:11,437::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1781::DEBUG::2012-08-30 12:57:11,437::task::978::TaskManager.Task::(_decref) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:57:13,235::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:15,260::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:15,261::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:18,236::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1783::DEBUG::2012-08-30 12:57:19,597::task::588::TaskManager.Task::(_updateState) Task=`868ee6c2-5800-413a-ad57-fa5c91732c90`::moving from state init -> state preparing >Thread-1783::INFO::2012-08-30 12:57:19,598::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1783::INFO::2012-08-30 12:57:19,598::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346302629.976028, 'code': 0, 'valid': True}} >Thread-1783::DEBUG::2012-08-30 12:57:19,598::task::1172::TaskManager.Task::(prepare) Task=`868ee6c2-5800-413a-ad57-fa5c91732c90`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346302629.976028, 'code': 0, 'valid': True}} >Thread-1783::DEBUG::2012-08-30 12:57:19,599::task::588::TaskManager.Task::(_updateState) Task=`868ee6c2-5800-413a-ad57-fa5c91732c90`::moving from state preparing -> state finished >Thread-1783::DEBUG::2012-08-30 12:57:19,599::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1783::DEBUG::2012-08-30 12:57:19,599::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1783::DEBUG::2012-08-30 12:57:19,599::task::978::TaskManager.Task::(_decref) Task=`868ee6c2-5800-413a-ad57-fa5c91732c90`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:57:20,262::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1784::DEBUG::2012-08-30 12:57:22,815::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1784::DEBUG::2012-08-30 12:57:22,815::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 12:57:23,237::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:23,237::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1785::DEBUG::2012-08-30 12:57:24,400::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1785::DEBUG::2012-08-30 12:57:24,401::task::588::TaskManager.Task::(_updateState) Task=`d12a60ad-197b-4576-9466-9d872315d6de`::moving from state init -> state preparing >Thread-1785::INFO::2012-08-30 12:57:24,401::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1785::INFO::2012-08-30 12:57:24,401::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1785::DEBUG::2012-08-30 12:57:24,401::task::1172::TaskManager.Task::(prepare) Task=`d12a60ad-197b-4576-9466-9d872315d6de`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1785::DEBUG::2012-08-30 12:57:24,402::task::588::TaskManager.Task::(_updateState) Task=`d12a60ad-197b-4576-9466-9d872315d6de`::moving from state preparing -> state finished >Thread-1785::DEBUG::2012-08-30 12:57:24,402::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1785::DEBUG::2012-08-30 12:57:24,402::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1785::DEBUG::2012-08-30 12:57:24,402::task::978::TaskManager.Task::(_decref) Task=`d12a60ad-197b-4576-9466-9d872315d6de`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:57:25,264::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:57:25,393::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1786::DEBUG::2012-08-30 12:57:27,604::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1786::DEBUG::2012-08-30 12:57:27,605::task::588::TaskManager.Task::(_updateState) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::moving from state init -> state preparing >Thread-1786::INFO::2012-08-30 12:57:27,605::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1786::DEBUG::2012-08-30 12:57:27,606::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`144b02af-5759-4605-9587-f791b0e92068`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1786::DEBUG::2012-08-30 12:57:27,606::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1786::DEBUG::2012-08-30 12:57:27,606::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1786::DEBUG::2012-08-30 12:57:27,607::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`144b02af-5759-4605-9587-f791b0e92068`::Granted request >Thread-1786::DEBUG::2012-08-30 12:57:27,607::task::817::TaskManager.Task::(resourceAcquired) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1786::DEBUG::2012-08-30 12:57:27,607::task::978::TaskManager.Task::(_decref) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::ref 1 aborting False >Thread-1786::INFO::2012-08-30 12:57:27,610::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1786::DEBUG::2012-08-30 12:57:27,610::task::1172::TaskManager.Task::(prepare) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1786::DEBUG::2012-08-30 12:57:27,611::task::588::TaskManager.Task::(_updateState) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::moving from state preparing -> state finished >Thread-1786::DEBUG::2012-08-30 12:57:27,611::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1786::DEBUG::2012-08-30 12:57:27,611::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1786::DEBUG::2012-08-30 12:57:27,611::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1786::DEBUG::2012-08-30 12:57:27,612::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1786::DEBUG::2012-08-30 12:57:27,612::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1786::DEBUG::2012-08-30 12:57:27,612::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1786::DEBUG::2012-08-30 12:57:27,612::task::978::TaskManager.Task::(_decref) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:57:28,238::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:57:30,265::task::588::TaskManager.Task::(_updateState) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:57:30,265::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:57:30,265::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c0c44eef-913f-44ee-977d-85e337da99f0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:57:30,266::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:57:30,266::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:57:30,266::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c0c44eef-913f-44ee-977d-85e337da99f0`::Granted request >Thread-104::DEBUG::2012-08-30 12:57:30,267::task::817::TaskManager.Task::(resourceAcquired) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:57:30,267::task::978::TaskManager.Task::(_decref) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:57:30,268::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:57:30,270::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:57:30,271::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:57:30,272::task::1172::TaskManager.Task::(prepare) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:57:30,272::task::588::TaskManager.Task::(_updateState) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:57:30,272::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:57:30,272::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:57:30,273::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:57:30,273::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:57:30,273::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:57:30,273::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:57:30,274::task::978::TaskManager.Task::(_decref) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:57:30,274::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:30,274::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:30,275::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:30,275::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:57:32,400::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:57:33,239::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:35,276::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:38,240::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:38,241::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:40,278::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1790::DEBUG::2012-08-30 12:57:40,834::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1790::DEBUG::2012-08-30 12:57:40,834::task::588::TaskManager.Task::(_updateState) Task=`4f5f6d99-52b0-4dce-b402-e4e111883cb9`::moving from state init -> state preparing >Thread-1790::INFO::2012-08-30 12:57:40,835::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1790::INFO::2012-08-30 12:57:40,835::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1790::DEBUG::2012-08-30 12:57:40,835::task::1172::TaskManager.Task::(prepare) Task=`4f5f6d99-52b0-4dce-b402-e4e111883cb9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1790::DEBUG::2012-08-30 12:57:40,835::task::588::TaskManager.Task::(_updateState) Task=`4f5f6d99-52b0-4dce-b402-e4e111883cb9`::moving from state preparing -> state finished >Thread-1790::DEBUG::2012-08-30 12:57:40,836::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1790::DEBUG::2012-08-30 12:57:40,836::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1790::DEBUG::2012-08-30 12:57:40,836::task::978::TaskManager.Task::(_decref) Task=`4f5f6d99-52b0-4dce-b402-e4e111883cb9`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:57:43,242::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1792::DEBUG::2012-08-30 12:57:44,021::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1792::DEBUG::2012-08-30 12:57:44,021::task::588::TaskManager.Task::(_updateState) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::moving from state init -> state preparing >Thread-1792::INFO::2012-08-30 12:57:44,021::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1792::DEBUG::2012-08-30 12:57:44,022::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a9ff7bba-a9c1-4d7f-8177-0b6ebdc53094`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1792::DEBUG::2012-08-30 12:57:44,022::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1792::DEBUG::2012-08-30 12:57:44,022::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1792::DEBUG::2012-08-30 12:57:44,023::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a9ff7bba-a9c1-4d7f-8177-0b6ebdc53094`::Granted request >Thread-1792::DEBUG::2012-08-30 12:57:44,023::task::817::TaskManager.Task::(resourceAcquired) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1792::DEBUG::2012-08-30 12:57:44,023::task::978::TaskManager.Task::(_decref) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::ref 1 aborting False >Thread-1792::INFO::2012-08-30 12:57:44,026::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1792::DEBUG::2012-08-30 12:57:44,026::task::1172::TaskManager.Task::(prepare) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1792::DEBUG::2012-08-30 12:57:44,027::task::588::TaskManager.Task::(_updateState) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::moving from state preparing -> state finished >Thread-1792::DEBUG::2012-08-30 12:57:44,027::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1792::DEBUG::2012-08-30 12:57:44,027::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1792::DEBUG::2012-08-30 12:57:44,027::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1792::DEBUG::2012-08-30 12:57:44,028::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1792::DEBUG::2012-08-30 12:57:44,028::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1792::DEBUG::2012-08-30 12:57:44,028::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1792::DEBUG::2012-08-30 12:57:44,028::task::978::TaskManager.Task::(_decref) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:57:45,279::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:45,279::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:48,242::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1793::DEBUG::2012-08-30 12:57:48,392::task::588::TaskManager.Task::(_updateState) Task=`ae121905-b38d-4412-bf49-ac4481e00b60`::moving from state init -> state preparing >Thread-1793::INFO::2012-08-30 12:57:48,392::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1793::INFO::2012-08-30 12:57:48,392::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346302659.9950261, 'code': 0, 'valid': True}} >Thread-1793::DEBUG::2012-08-30 12:57:48,392::task::1172::TaskManager.Task::(prepare) Task=`ae121905-b38d-4412-bf49-ac4481e00b60`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346302659.9950261, 'code': 0, 'valid': True}} >Thread-1793::DEBUG::2012-08-30 12:57:48,393::task::588::TaskManager.Task::(_updateState) Task=`ae121905-b38d-4412-bf49-ac4481e00b60`::moving from state preparing -> state finished >Thread-1793::DEBUG::2012-08-30 12:57:48,393::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1793::DEBUG::2012-08-30 12:57:48,393::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1793::DEBUG::2012-08-30 12:57:48,393::task::978::TaskManager.Task::(_decref) Task=`ae121905-b38d-4412-bf49-ac4481e00b60`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:57:50,281::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1794::DEBUG::2012-08-30 12:57:51,579::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1794::DEBUG::2012-08-30 12:57:51,580::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::DEBUG::2012-08-30 12:57:53,243::task::588::TaskManager.Task::(_updateState) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:57:53,244::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:57:53,244::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`1caa370f-00d7-4111-a6f9-77bb9b0ccee1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:57:53,244::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:57:53,245::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:57:53,245::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`1caa370f-00d7-4111-a6f9-77bb9b0ccee1`::Granted request >Thread-40::DEBUG::2012-08-30 12:57:53,245::task::817::TaskManager.Task::(resourceAcquired) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:57:53,245::task::978::TaskManager.Task::(_decref) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:57:53,247::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:57:53,249::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:57:53,250::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:57:53,251::task::1172::TaskManager.Task::(prepare) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:57:53,251::task::588::TaskManager.Task::(_updateState) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:57:53,251::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:57:53,251::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:57:53,252::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:57:53,252::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:57:53,252::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:57:53,252::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:57:53,253::task::978::TaskManager.Task::(_decref) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:57:53,253::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:53,253::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:53,254::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:57:53,254::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:57:55,282::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:57:55,421::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1796::DEBUG::2012-08-30 12:57:56,843::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1796::DEBUG::2012-08-30 12:57:56,844::task::588::TaskManager.Task::(_updateState) Task=`2bf4d963-11a0-4eab-9c35-a4248fdc28b8`::moving from state init -> state preparing >Thread-1796::INFO::2012-08-30 12:57:56,844::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1796::INFO::2012-08-30 12:57:56,844::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1796::DEBUG::2012-08-30 12:57:56,845::task::1172::TaskManager.Task::(prepare) Task=`2bf4d963-11a0-4eab-9c35-a4248fdc28b8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1796::DEBUG::2012-08-30 12:57:56,845::task::588::TaskManager.Task::(_updateState) Task=`2bf4d963-11a0-4eab-9c35-a4248fdc28b8`::moving from state preparing -> state finished >Thread-1796::DEBUG::2012-08-30 12:57:56,845::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1796::DEBUG::2012-08-30 12:57:56,845::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1796::DEBUG::2012-08-30 12:57:56,846::task::978::TaskManager.Task::(_decref) Task=`2bf4d963-11a0-4eab-9c35-a4248fdc28b8`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:57:58,255::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1797::DEBUG::2012-08-30 12:58:00,056::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1797::DEBUG::2012-08-30 12:58:00,056::task::588::TaskManager.Task::(_updateState) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::moving from state init -> state preparing >Thread-1797::INFO::2012-08-30 12:58:00,056::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1797::DEBUG::2012-08-30 12:58:00,057::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e31b2058-7129-4bd5-9d0d-921a8ceba4ed`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1797::DEBUG::2012-08-30 12:58:00,057::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1797::DEBUG::2012-08-30 12:58:00,057::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1797::DEBUG::2012-08-30 12:58:00,058::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e31b2058-7129-4bd5-9d0d-921a8ceba4ed`::Granted request >Thread-1797::DEBUG::2012-08-30 12:58:00,058::task::817::TaskManager.Task::(resourceAcquired) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1797::DEBUG::2012-08-30 12:58:00,058::task::978::TaskManager.Task::(_decref) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::ref 1 aborting False >Thread-1797::INFO::2012-08-30 12:58:00,061::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1797::DEBUG::2012-08-30 12:58:00,061::task::1172::TaskManager.Task::(prepare) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1797::DEBUG::2012-08-30 12:58:00,062::task::588::TaskManager.Task::(_updateState) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::moving from state preparing -> state finished >Thread-1797::DEBUG::2012-08-30 12:58:00,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1797::DEBUG::2012-08-30 12:58:00,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1797::DEBUG::2012-08-30 12:58:00,062::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1797::DEBUG::2012-08-30 12:58:00,063::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1797::DEBUG::2012-08-30 12:58:00,063::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1797::DEBUG::2012-08-30 12:58:00,063::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1797::DEBUG::2012-08-30 12:58:00,063::task::978::TaskManager.Task::(_decref) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:58:00,283::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:00,284::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:58:02,426::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:58:03,256::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:05,285::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:08,257::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:08,257::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:10,286::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:13,258::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1801::DEBUG::2012-08-30 12:58:14,091::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1801::DEBUG::2012-08-30 12:58:14,092::task::588::TaskManager.Task::(_updateState) Task=`35edd4b5-6c63-4968-9c0c-a6412197afe6`::moving from state init -> state preparing >Thread-1801::INFO::2012-08-30 12:58:14,092::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1801::INFO::2012-08-30 12:58:14,092::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1801::DEBUG::2012-08-30 12:58:14,092::task::1172::TaskManager.Task::(prepare) Task=`35edd4b5-6c63-4968-9c0c-a6412197afe6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1801::DEBUG::2012-08-30 12:58:14,093::task::588::TaskManager.Task::(_updateState) Task=`35edd4b5-6c63-4968-9c0c-a6412197afe6`::moving from state preparing -> state finished >Thread-1801::DEBUG::2012-08-30 12:58:14,093::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1801::DEBUG::2012-08-30 12:58:14,093::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1801::DEBUG::2012-08-30 12:58:14,093::task::978::TaskManager.Task::(_decref) Task=`35edd4b5-6c63-4968-9c0c-a6412197afe6`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:58:15,287::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:15,288::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1802::DEBUG::2012-08-30 12:58:17,305::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1802::DEBUG::2012-08-30 12:58:17,305::task::588::TaskManager.Task::(_updateState) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::moving from state init -> state preparing >Thread-1802::INFO::2012-08-30 12:58:17,305::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1802::DEBUG::2012-08-30 12:58:17,306::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`955d82f5-8155-424d-bce7-b2bb3772af5d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1802::DEBUG::2012-08-30 12:58:17,306::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1802::DEBUG::2012-08-30 12:58:17,306::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1802::DEBUG::2012-08-30 12:58:17,306::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`955d82f5-8155-424d-bce7-b2bb3772af5d`::Granted request >Thread-1802::DEBUG::2012-08-30 12:58:17,307::task::817::TaskManager.Task::(resourceAcquired) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1802::DEBUG::2012-08-30 12:58:17,307::task::978::TaskManager.Task::(_decref) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::ref 1 aborting False >Thread-1802::INFO::2012-08-30 12:58:17,310::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1802::DEBUG::2012-08-30 12:58:17,310::task::1172::TaskManager.Task::(prepare) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1802::DEBUG::2012-08-30 12:58:17,310::task::588::TaskManager.Task::(_updateState) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::moving from state preparing -> state finished >Thread-1802::DEBUG::2012-08-30 12:58:17,311::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1802::DEBUG::2012-08-30 12:58:17,311::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1802::DEBUG::2012-08-30 12:58:17,311::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1802::DEBUG::2012-08-30 12:58:17,311::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1802::DEBUG::2012-08-30 12:58:17,312::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1802::DEBUG::2012-08-30 12:58:17,312::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1802::DEBUG::2012-08-30 12:58:17,312::task::978::TaskManager.Task::(_decref) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:58:18,259::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1803::DEBUG::2012-08-30 12:58:18,918::task::588::TaskManager.Task::(_updateState) Task=`bd6a93ac-948b-4bf1-911a-947fef35d8ae`::moving from state init -> state preparing >Thread-1803::INFO::2012-08-30 12:58:18,918::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1803::INFO::2012-08-30 12:58:18,918::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346302690.0157461, 'code': 0, 'valid': True}} >Thread-1803::DEBUG::2012-08-30 12:58:18,919::task::1172::TaskManager.Task::(prepare) Task=`bd6a93ac-948b-4bf1-911a-947fef35d8ae`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346302690.0157461, 'code': 0, 'valid': True}} >Thread-1803::DEBUG::2012-08-30 12:58:18,919::task::588::TaskManager.Task::(_updateState) Task=`bd6a93ac-948b-4bf1-911a-947fef35d8ae`::moving from state preparing -> state finished >Thread-1803::DEBUG::2012-08-30 12:58:18,919::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1803::DEBUG::2012-08-30 12:58:18,919::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1803::DEBUG::2012-08-30 12:58:18,920::task::978::TaskManager.Task::(_decref) Task=`bd6a93ac-948b-4bf1-911a-947fef35d8ae`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:58:20,289::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1804::DEBUG::2012-08-30 12:58:22,224::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1804::DEBUG::2012-08-30 12:58:22,225::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 12:58:23,260::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:23,261::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:25,290::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:58:25,441::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 12:58:28,262::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:58:30,291::task::588::TaskManager.Task::(_updateState) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:58:30,292::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:58:30,292::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`8afa8b36-941c-47fc-b3d6-4044438c542f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:58:30,292::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:58:30,293::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:58:30,293::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`8afa8b36-941c-47fc-b3d6-4044438c542f`::Granted request >Thread-104::DEBUG::2012-08-30 12:58:30,293::task::817::TaskManager.Task::(resourceAcquired) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:58:30,294::task::978::TaskManager.Task::(_decref) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:58:30,295::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:58:30,297::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:58:30,298::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:58:30,298::task::1172::TaskManager.Task::(prepare) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:58:30,299::task::588::TaskManager.Task::(_updateState) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:58:30,299::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:58:30,299::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:58:30,299::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:58:30,300::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:58:30,300::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:58:30,300::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:58:30,300::task::978::TaskManager.Task::(_decref) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:58:30,301::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:30,301::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:30,302::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:30,302::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:58:32,447::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1806::DEBUG::2012-08-30 12:58:32,729::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1806::DEBUG::2012-08-30 12:58:32,729::task::588::TaskManager.Task::(_updateState) Task=`b06b735b-8adf-44a6-a7b1-1a1b377ad2b1`::moving from state init -> state preparing >Thread-1806::INFO::2012-08-30 12:58:32,729::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1806::INFO::2012-08-30 12:58:32,730::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1806::DEBUG::2012-08-30 12:58:32,730::task::1172::TaskManager.Task::(prepare) Task=`b06b735b-8adf-44a6-a7b1-1a1b377ad2b1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1806::DEBUG::2012-08-30 12:58:32,730::task::588::TaskManager.Task::(_updateState) Task=`b06b735b-8adf-44a6-a7b1-1a1b377ad2b1`::moving from state preparing -> state finished >Thread-1806::DEBUG::2012-08-30 12:58:32,730::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1806::DEBUG::2012-08-30 12:58:32,731::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1806::DEBUG::2012-08-30 12:58:32,731::task::978::TaskManager.Task::(_decref) Task=`b06b735b-8adf-44a6-a7b1-1a1b377ad2b1`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:58:33,262::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:35,303::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1808::DEBUG::2012-08-30 12:58:36,039::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1808::DEBUG::2012-08-30 12:58:36,039::task::588::TaskManager.Task::(_updateState) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::moving from state init -> state preparing >Thread-1808::INFO::2012-08-30 12:58:36,039::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1808::DEBUG::2012-08-30 12:58:36,040::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ff7c745a-8243-4ee5-9ae6-6c338fff913d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1808::DEBUG::2012-08-30 12:58:36,040::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1808::DEBUG::2012-08-30 12:58:36,040::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1808::DEBUG::2012-08-30 12:58:36,040::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ff7c745a-8243-4ee5-9ae6-6c338fff913d`::Granted request >Thread-1808::DEBUG::2012-08-30 12:58:36,041::task::817::TaskManager.Task::(resourceAcquired) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1808::DEBUG::2012-08-30 12:58:36,041::task::978::TaskManager.Task::(_decref) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::ref 1 aborting False >Thread-1808::INFO::2012-08-30 12:58:36,044::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1808::DEBUG::2012-08-30 12:58:36,044::task::1172::TaskManager.Task::(prepare) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1808::DEBUG::2012-08-30 12:58:36,045::task::588::TaskManager.Task::(_updateState) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::moving from state preparing -> state finished >Thread-1808::DEBUG::2012-08-30 12:58:36,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1808::DEBUG::2012-08-30 12:58:36,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1808::DEBUG::2012-08-30 12:58:36,045::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1808::DEBUG::2012-08-30 12:58:36,045::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1808::DEBUG::2012-08-30 12:58:36,046::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1808::DEBUG::2012-08-30 12:58:36,046::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1808::DEBUG::2012-08-30 12:58:36,046::task::978::TaskManager.Task::(_decref) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:58:38,263::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:38,264::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:40,304::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:43,265::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:45,306::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:58:45,306::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:48,266::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1811::DEBUG::2012-08-30 12:58:49,298::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1811::DEBUG::2012-08-30 12:58:49,299::task::588::TaskManager.Task::(_updateState) Task=`0714c63d-b461-4dd6-8d4b-1ee0599a14ce`::moving from state init -> state preparing >Thread-1811::INFO::2012-08-30 12:58:49,299::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1811::INFO::2012-08-30 12:58:49,299::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1811::DEBUG::2012-08-30 12:58:49,300::task::1172::TaskManager.Task::(prepare) Task=`0714c63d-b461-4dd6-8d4b-1ee0599a14ce`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1811::DEBUG::2012-08-30 12:58:49,300::task::588::TaskManager.Task::(_updateState) Task=`0714c63d-b461-4dd6-8d4b-1ee0599a14ce`::moving from state preparing -> state finished >Thread-1811::DEBUG::2012-08-30 12:58:49,300::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1811::DEBUG::2012-08-30 12:58:49,300::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1811::DEBUG::2012-08-30 12:58:49,301::task::978::TaskManager.Task::(_decref) Task=`0714c63d-b461-4dd6-8d4b-1ee0599a14ce`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:58:50,308::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1812::DEBUG::2012-08-30 12:58:50,317::task::588::TaskManager.Task::(_updateState) Task=`2d711989-8508-45a0-be85-8648f76f13fd`::moving from state init -> state preparing >Thread-1812::INFO::2012-08-30 12:58:50,317::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1812::INFO::2012-08-30 12:58:50,317::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176596641541', 'lastCheck': 1346302730.0416851, 'code': 0, 'valid': True}} >Thread-1812::DEBUG::2012-08-30 12:58:50,318::task::1172::TaskManager.Task::(prepare) Task=`2d711989-8508-45a0-be85-8648f76f13fd`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176596641541', 'lastCheck': 1346302730.0416851, 'code': 0, 'valid': True}} >Thread-1812::DEBUG::2012-08-30 12:58:50,318::task::588::TaskManager.Task::(_updateState) Task=`2d711989-8508-45a0-be85-8648f76f13fd`::moving from state preparing -> state finished >Thread-1812::DEBUG::2012-08-30 12:58:50,318::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1812::DEBUG::2012-08-30 12:58:50,318::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1812::DEBUG::2012-08-30 12:58:50,319::task::978::TaskManager.Task::(_decref) Task=`2d711989-8508-45a0-be85-8648f76f13fd`::ref 0 aborting False >Thread-1813::DEBUG::2012-08-30 12:58:51,930::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1813::DEBUG::2012-08-30 12:58:51,930::task::588::TaskManager.Task::(_updateState) Task=`82697733-fdf1-49b5-9001-cf605785541e`::moving from state init -> state preparing >Thread-1813::INFO::2012-08-30 12:58:51,930::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1813::DEBUG::2012-08-30 12:58:51,931::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6ed520ee-e823-45ff-857a-d43cd760cbef`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1813::DEBUG::2012-08-30 12:58:51,931::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1813::DEBUG::2012-08-30 12:58:51,931::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1813::DEBUG::2012-08-30 12:58:51,931::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6ed520ee-e823-45ff-857a-d43cd760cbef`::Granted request >Thread-1813::DEBUG::2012-08-30 12:58:51,932::task::817::TaskManager.Task::(resourceAcquired) Task=`82697733-fdf1-49b5-9001-cf605785541e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1813::DEBUG::2012-08-30 12:58:51,932::task::978::TaskManager.Task::(_decref) Task=`82697733-fdf1-49b5-9001-cf605785541e`::ref 1 aborting False >Thread-1813::INFO::2012-08-30 12:58:51,935::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1813::DEBUG::2012-08-30 12:58:51,935::task::1172::TaskManager.Task::(prepare) Task=`82697733-fdf1-49b5-9001-cf605785541e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1813::DEBUG::2012-08-30 12:58:51,935::task::588::TaskManager.Task::(_updateState) Task=`82697733-fdf1-49b5-9001-cf605785541e`::moving from state preparing -> state finished >Thread-1813::DEBUG::2012-08-30 12:58:51,936::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1813::DEBUG::2012-08-30 12:58:51,936::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1813::DEBUG::2012-08-30 12:58:51,936::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1813::DEBUG::2012-08-30 12:58:51,936::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1813::DEBUG::2012-08-30 12:58:51,937::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1813::DEBUG::2012-08-30 12:58:51,937::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1813::DEBUG::2012-08-30 12:58:51,937::task::978::TaskManager.Task::(_decref) Task=`82697733-fdf1-49b5-9001-cf605785541e`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 12:58:53,267::task::588::TaskManager.Task::(_updateState) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:58:53,267::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:58:53,267::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bc26458a-3f7a-42dc-ba5e-60be4cb5fb6f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:58:53,268::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:58:53,268::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:58:53,268::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bc26458a-3f7a-42dc-ba5e-60be4cb5fb6f`::Granted request >Thread-40::DEBUG::2012-08-30 12:58:53,269::task::817::TaskManager.Task::(resourceAcquired) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:58:53,269::task::978::TaskManager.Task::(_decref) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:58:53,271::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:58:53,273::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:58:53,274::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:58:53,274::task::1172::TaskManager.Task::(prepare) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:58:53,274::task::588::TaskManager.Task::(_updateState) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:58:53,274::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:58:53,275::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:58:53,275::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:58:53,275::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:58:53,275::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:58:53,276::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:58:53,276::task::978::TaskManager.Task::(_decref) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:58:53,276::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:53,277::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:53,277::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:58:53,278::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1814::DEBUG::2012-08-30 12:58:53,523::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1814::DEBUG::2012-08-30 12:58:53,524::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 12:58:55,309::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:58:55,468::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 12:58:58,278::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:00,310::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:00,311::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:59:02,474::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:59:03,279::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1817::DEBUG::2012-08-30 12:59:05,162::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1817::DEBUG::2012-08-30 12:59:05,163::task::588::TaskManager.Task::(_updateState) Task=`ecc9be1d-6183-4f61-993f-615bb79036d0`::moving from state init -> state preparing >Thread-1817::INFO::2012-08-30 12:59:05,163::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1817::INFO::2012-08-30 12:59:05,163::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1817::DEBUG::2012-08-30 12:59:05,163::task::1172::TaskManager.Task::(prepare) Task=`ecc9be1d-6183-4f61-993f-615bb79036d0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1817::DEBUG::2012-08-30 12:59:05,164::task::588::TaskManager.Task::(_updateState) Task=`ecc9be1d-6183-4f61-993f-615bb79036d0`::moving from state preparing -> state finished >Thread-1817::DEBUG::2012-08-30 12:59:05,164::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1817::DEBUG::2012-08-30 12:59:05,164::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1817::DEBUG::2012-08-30 12:59:05,164::task::978::TaskManager.Task::(_decref) Task=`ecc9be1d-6183-4f61-993f-615bb79036d0`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:59:05,312::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1818::DEBUG::2012-08-30 12:59:08,101::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1818::DEBUG::2012-08-30 12:59:08,101::task::588::TaskManager.Task::(_updateState) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::moving from state init -> state preparing >Thread-1818::INFO::2012-08-30 12:59:08,102::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1818::DEBUG::2012-08-30 12:59:08,102::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0f5d3e56-fdf7-437a-900f-01c2f5c8537c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1818::DEBUG::2012-08-30 12:59:08,102::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1818::DEBUG::2012-08-30 12:59:08,102::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1818::DEBUG::2012-08-30 12:59:08,103::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0f5d3e56-fdf7-437a-900f-01c2f5c8537c`::Granted request >Thread-1818::DEBUG::2012-08-30 12:59:08,103::task::817::TaskManager.Task::(resourceAcquired) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1818::DEBUG::2012-08-30 12:59:08,103::task::978::TaskManager.Task::(_decref) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::ref 1 aborting False >Thread-1818::INFO::2012-08-30 12:59:08,106::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1818::DEBUG::2012-08-30 12:59:08,106::task::1172::TaskManager.Task::(prepare) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1818::DEBUG::2012-08-30 12:59:08,107::task::588::TaskManager.Task::(_updateState) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::moving from state preparing -> state finished >Thread-1818::DEBUG::2012-08-30 12:59:08,107::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1818::DEBUG::2012-08-30 12:59:08,107::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1818::DEBUG::2012-08-30 12:59:08,107::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1818::DEBUG::2012-08-30 12:59:08,108::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1818::DEBUG::2012-08-30 12:59:08,108::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1818::DEBUG::2012-08-30 12:59:08,108::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1818::DEBUG::2012-08-30 12:59:08,108::task::978::TaskManager.Task::(_decref) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:59:08,280::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:08,281::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:10,313::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:13,282::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:15,314::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:15,315::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:18,283::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:20,316::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1821::DEBUG::2012-08-30 12:59:20,737::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1821::DEBUG::2012-08-30 12:59:20,738::task::588::TaskManager.Task::(_updateState) Task=`8719494b-6d80-48f8-beaf-07c84be5f7b2`::moving from state init -> state preparing >Thread-1821::INFO::2012-08-30 12:59:20,738::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1821::INFO::2012-08-30 12:59:20,738::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1821::DEBUG::2012-08-30 12:59:20,738::task::1172::TaskManager.Task::(prepare) Task=`8719494b-6d80-48f8-beaf-07c84be5f7b2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1821::DEBUG::2012-08-30 12:59:20,739::task::588::TaskManager.Task::(_updateState) Task=`8719494b-6d80-48f8-beaf-07c84be5f7b2`::moving from state preparing -> state finished >Thread-1821::DEBUG::2012-08-30 12:59:20,739::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1821::DEBUG::2012-08-30 12:59:20,739::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1821::DEBUG::2012-08-30 12:59:20,739::task::978::TaskManager.Task::(_decref) Task=`8719494b-6d80-48f8-beaf-07c84be5f7b2`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:59:23,284::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:23,284::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1822::DEBUG::2012-08-30 12:59:24,359::task::588::TaskManager.Task::(_updateState) Task=`7f6212e2-a8ff-4354-9ad9-607c734b4f06`::moving from state init -> state preparing >Thread-1822::INFO::2012-08-30 12:59:24,359::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1822::INFO::2012-08-30 12:59:24,360::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164389610291', 'lastCheck': 1346302760.061686, 'code': 0, 'valid': True}} >Thread-1822::DEBUG::2012-08-30 12:59:24,360::task::1172::TaskManager.Task::(prepare) Task=`7f6212e2-a8ff-4354-9ad9-607c734b4f06`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164389610291', 'lastCheck': 1346302760.061686, 'code': 0, 'valid': True}} >Thread-1822::DEBUG::2012-08-30 12:59:24,360::task::588::TaskManager.Task::(_updateState) Task=`7f6212e2-a8ff-4354-9ad9-607c734b4f06`::moving from state preparing -> state finished >Thread-1822::DEBUG::2012-08-30 12:59:24,360::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1822::DEBUG::2012-08-30 12:59:24,361::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1822::DEBUG::2012-08-30 12:59:24,361::task::978::TaskManager.Task::(_decref) Task=`7f6212e2-a8ff-4354-9ad9-607c734b4f06`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:59:25,317::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:59:25,491::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1823::DEBUG::2012-08-30 12:59:26,039::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1823::DEBUG::2012-08-30 12:59:26,040::task::588::TaskManager.Task::(_updateState) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::moving from state init -> state preparing >Thread-1823::INFO::2012-08-30 12:59:26,040::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1823::DEBUG::2012-08-30 12:59:26,040::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`4fa24500-3f49-4a18-830d-0eda438d7c80`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1823::DEBUG::2012-08-30 12:59:26,041::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1823::DEBUG::2012-08-30 12:59:26,041::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1823::DEBUG::2012-08-30 12:59:26,041::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`4fa24500-3f49-4a18-830d-0eda438d7c80`::Granted request >Thread-1823::DEBUG::2012-08-30 12:59:26,041::task::817::TaskManager.Task::(resourceAcquired) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1823::DEBUG::2012-08-30 12:59:26,042::task::978::TaskManager.Task::(_decref) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::ref 1 aborting False >Thread-1823::INFO::2012-08-30 12:59:26,045::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1823::DEBUG::2012-08-30 12:59:26,045::task::1172::TaskManager.Task::(prepare) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1823::DEBUG::2012-08-30 12:59:26,045::task::588::TaskManager.Task::(_updateState) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::moving from state preparing -> state finished >Thread-1823::DEBUG::2012-08-30 12:59:26,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1823::DEBUG::2012-08-30 12:59:26,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1823::DEBUG::2012-08-30 12:59:26,046::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1823::DEBUG::2012-08-30 12:59:26,046::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1823::DEBUG::2012-08-30 12:59:26,046::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1823::DEBUG::2012-08-30 12:59:26,046::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1823::DEBUG::2012-08-30 12:59:26,047::task::978::TaskManager.Task::(_decref) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::ref 0 aborting False >Thread-1824::DEBUG::2012-08-30 12:59:27,631::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1824::DEBUG::2012-08-30 12:59:27,632::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 12:59:28,285::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 12:59:30,318::task::588::TaskManager.Task::(_updateState) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 12:59:30,318::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 12:59:30,319::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`693d203e-ad8b-4981-87e7-772383b37c7d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 12:59:30,319::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 12:59:30,319::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 12:59:30,320::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`693d203e-ad8b-4981-87e7-772383b37c7d`::Granted request >Thread-104::DEBUG::2012-08-30 12:59:30,320::task::817::TaskManager.Task::(resourceAcquired) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 12:59:30,320::task::978::TaskManager.Task::(_decref) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 12:59:30,322::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 12:59:30,324::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 12:59:30,325::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:59:30,325::task::1172::TaskManager.Task::(prepare) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 12:59:30,325::task::588::TaskManager.Task::(_updateState) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 12:59:30,326::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 12:59:30,326::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 12:59:30,326::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 12:59:30,326::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 12:59:30,327::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 12:59:30,327::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 12:59:30,327::task::978::TaskManager.Task::(_decref) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:59:30,327::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:30,328::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:30,328::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:30,329::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:59:32,498::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 12:59:33,286::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:35,330::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:38,287::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:38,288::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1827::DEBUG::2012-08-30 12:59:39,307::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1827::DEBUG::2012-08-30 12:59:39,307::task::588::TaskManager.Task::(_updateState) Task=`61139ffc-49c5-4e66-80f3-a1897d203d26`::moving from state init -> state preparing >Thread-1827::INFO::2012-08-30 12:59:39,307::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1827::INFO::2012-08-30 12:59:39,308::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1827::DEBUG::2012-08-30 12:59:39,308::task::1172::TaskManager.Task::(prepare) Task=`61139ffc-49c5-4e66-80f3-a1897d203d26`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1827::DEBUG::2012-08-30 12:59:39,308::task::588::TaskManager.Task::(_updateState) Task=`61139ffc-49c5-4e66-80f3-a1897d203d26`::moving from state preparing -> state finished >Thread-1827::DEBUG::2012-08-30 12:59:39,308::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1827::DEBUG::2012-08-30 12:59:39,309::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1827::DEBUG::2012-08-30 12:59:39,309::task::978::TaskManager.Task::(_decref) Task=`61139ffc-49c5-4e66-80f3-a1897d203d26`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:59:40,331::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1828::DEBUG::2012-08-30 12:59:42,579::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1828::DEBUG::2012-08-30 12:59:42,580::task::588::TaskManager.Task::(_updateState) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::moving from state init -> state preparing >Thread-1828::INFO::2012-08-30 12:59:42,580::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1828::DEBUG::2012-08-30 12:59:42,580::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d83eb4a9-2468-437f-85cf-3dfc51876d67`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1828::DEBUG::2012-08-30 12:59:42,581::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1828::DEBUG::2012-08-30 12:59:42,581::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1828::DEBUG::2012-08-30 12:59:42,581::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d83eb4a9-2468-437f-85cf-3dfc51876d67`::Granted request >Thread-1828::DEBUG::2012-08-30 12:59:42,581::task::817::TaskManager.Task::(resourceAcquired) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1828::DEBUG::2012-08-30 12:59:42,582::task::978::TaskManager.Task::(_decref) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::ref 1 aborting False >Thread-1828::INFO::2012-08-30 12:59:42,585::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1828::DEBUG::2012-08-30 12:59:42,585::task::1172::TaskManager.Task::(prepare) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1828::DEBUG::2012-08-30 12:59:42,585::task::588::TaskManager.Task::(_updateState) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::moving from state preparing -> state finished >Thread-1828::DEBUG::2012-08-30 12:59:42,585::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1828::DEBUG::2012-08-30 12:59:42,587::task::978::TaskManager.Task::(_decref) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:59:43,288::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:45,332::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:45,333::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:48,289::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 12:59:50,334::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 12:59:53,290::task::588::TaskManager.Task::(_updateState) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 12:59:53,291::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 12:59:53,291::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c90d95b1-46e3-4fec-82fd-345a70934052`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 12:59:53,291::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 12:59:53,291::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 12:59:53,292::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c90d95b1-46e3-4fec-82fd-345a70934052`::Granted request >Thread-40::DEBUG::2012-08-30 12:59:53,292::task::817::TaskManager.Task::(resourceAcquired) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 12:59:53,292::task::978::TaskManager.Task::(_decref) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 12:59:53,294::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 12:59:53,296::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 12:59:53,297::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:59:53,297::task::1172::TaskManager.Task::(prepare) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 12:59:53,298::task::588::TaskManager.Task::(_updateState) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 12:59:53,298::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 12:59:53,298::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 12:59:53,298::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 12:59:53,299::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 12:59:53,299::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 12:59:53,299::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 12:59:53,299::task::978::TaskManager.Task::(_decref) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 12:59:53,300::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:53,300::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:53,301::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 12:59:53,301::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1832::DEBUG::2012-08-30 12:59:55,288::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1831::DEBUG::2012-08-30 12:59:55,288::task::588::TaskManager.Task::(_updateState) Task=`d6d6894b-478a-4fbe-887c-6ef0d1b596a4`::moving from state init -> state preparing >Thread-1832::DEBUG::2012-08-30 12:59:55,288::task::588::TaskManager.Task::(_updateState) Task=`c23300cf-587a-409e-9ddf-e701fa98d4e8`::moving from state init -> state preparing >Thread-1831::INFO::2012-08-30 12:59:55,289::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1832::INFO::2012-08-30 12:59:55,289::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1831::INFO::2012-08-30 12:59:55,289::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0025589466095', 'lastCheck': 1346302790.081907, 'code': 0, 'valid': True}} >Thread-1832::INFO::2012-08-30 12:59:55,289::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1831::DEBUG::2012-08-30 12:59:55,290::task::1172::TaskManager.Task::(prepare) Task=`d6d6894b-478a-4fbe-887c-6ef0d1b596a4`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0025589466095', 'lastCheck': 1346302790.081907, 'code': 0, 'valid': True}} >Thread-1832::DEBUG::2012-08-30 12:59:55,290::task::1172::TaskManager.Task::(prepare) Task=`c23300cf-587a-409e-9ddf-e701fa98d4e8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1831::DEBUG::2012-08-30 12:59:55,290::task::588::TaskManager.Task::(_updateState) Task=`d6d6894b-478a-4fbe-887c-6ef0d1b596a4`::moving from state preparing -> state finished >Thread-1832::DEBUG::2012-08-30 12:59:55,290::task::588::TaskManager.Task::(_updateState) Task=`c23300cf-587a-409e-9ddf-e701fa98d4e8`::moving from state preparing -> state finished >Thread-1831::DEBUG::2012-08-30 12:59:55,290::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1832::DEBUG::2012-08-30 12:59:55,291::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1831::DEBUG::2012-08-30 12:59:55,291::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1832::DEBUG::2012-08-30 12:59:55,291::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1831::DEBUG::2012-08-30 12:59:55,291::task::978::TaskManager.Task::(_decref) Task=`d6d6894b-478a-4fbe-887c-6ef0d1b596a4`::ref 0 aborting False >Thread-1832::DEBUG::2012-08-30 12:59:55,291::task::978::TaskManager.Task::(_decref) Task=`c23300cf-587a-409e-9ddf-e701fa98d4e8`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 12:59:55,335::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 12:59:55,512::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 12:59:58,302::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1834::DEBUG::2012-08-30 12:59:58,496::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1834::DEBUG::2012-08-30 12:59:58,496::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1833::DEBUG::2012-08-30 12:59:58,499::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1833::DEBUG::2012-08-30 12:59:58,499::task::588::TaskManager.Task::(_updateState) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::moving from state init -> state preparing >Thread-1833::INFO::2012-08-30 12:59:58,499::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1833::DEBUG::2012-08-30 12:59:58,500::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f07bb43d-fe60-45f0-ab6d-919a72eff272`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1833::DEBUG::2012-08-30 12:59:58,500::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1833::DEBUG::2012-08-30 12:59:58,500::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1833::DEBUG::2012-08-30 12:59:58,500::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f07bb43d-fe60-45f0-ab6d-919a72eff272`::Granted request >Thread-1833::DEBUG::2012-08-30 12:59:58,501::task::817::TaskManager.Task::(resourceAcquired) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1833::DEBUG::2012-08-30 12:59:58,501::task::978::TaskManager.Task::(_decref) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::ref 1 aborting False >Thread-1833::INFO::2012-08-30 12:59:58,504::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1833::DEBUG::2012-08-30 12:59:58,504::task::1172::TaskManager.Task::(prepare) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1833::DEBUG::2012-08-30 12:59:58,504::task::588::TaskManager.Task::(_updateState) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::moving from state preparing -> state finished >Thread-1833::DEBUG::2012-08-30 12:59:58,505::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1833::DEBUG::2012-08-30 12:59:58,505::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1833::DEBUG::2012-08-30 12:59:58,505::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1833::DEBUG::2012-08-30 12:59:58,505::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1833::DEBUG::2012-08-30 12:59:58,506::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1833::DEBUG::2012-08-30 12:59:58,506::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1833::DEBUG::2012-08-30 12:59:58,506::task::978::TaskManager.Task::(_decref) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:00:00,337::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:00,337::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:00:02,520::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:00:03,303::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:05,338::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:08,304::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:08,304::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:10,339::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1837::DEBUG::2012-08-30 13:00:11,819::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1837::DEBUG::2012-08-30 13:00:11,819::task::588::TaskManager.Task::(_updateState) Task=`dcf03c10-d1ec-4f84-b6cb-cbec6b7735d9`::moving from state init -> state preparing >Thread-1837::INFO::2012-08-30 13:00:11,819::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1837::INFO::2012-08-30 13:00:11,820::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1837::DEBUG::2012-08-30 13:00:11,820::task::1172::TaskManager.Task::(prepare) Task=`dcf03c10-d1ec-4f84-b6cb-cbec6b7735d9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1837::DEBUG::2012-08-30 13:00:11,820::task::588::TaskManager.Task::(_updateState) Task=`dcf03c10-d1ec-4f84-b6cb-cbec6b7735d9`::moving from state preparing -> state finished >Thread-1837::DEBUG::2012-08-30 13:00:11,820::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1837::DEBUG::2012-08-30 13:00:11,820::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1837::DEBUG::2012-08-30 13:00:11,821::task::978::TaskManager.Task::(_decref) Task=`dcf03c10-d1ec-4f84-b6cb-cbec6b7735d9`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:00:13,305::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1839::DEBUG::2012-08-30 13:00:15,030::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1839::DEBUG::2012-08-30 13:00:15,030::task::588::TaskManager.Task::(_updateState) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::moving from state init -> state preparing >Thread-1839::INFO::2012-08-30 13:00:15,031::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1839::DEBUG::2012-08-30 13:00:15,031::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2054d869-d99d-4567-8134-83af18286b29`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1839::DEBUG::2012-08-30 13:00:15,031::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1839::DEBUG::2012-08-30 13:00:15,032::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1839::DEBUG::2012-08-30 13:00:15,032::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2054d869-d99d-4567-8134-83af18286b29`::Granted request >Thread-1839::DEBUG::2012-08-30 13:00:15,032::task::817::TaskManager.Task::(resourceAcquired) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1839::DEBUG::2012-08-30 13:00:15,032::task::978::TaskManager.Task::(_decref) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::ref 1 aborting False >Thread-1839::INFO::2012-08-30 13:00:15,035::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1839::DEBUG::2012-08-30 13:00:15,035::task::1172::TaskManager.Task::(prepare) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1839::DEBUG::2012-08-30 13:00:15,036::task::588::TaskManager.Task::(_updateState) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::moving from state preparing -> state finished >Thread-1839::DEBUG::2012-08-30 13:00:15,036::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1839::DEBUG::2012-08-30 13:00:15,036::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1839::DEBUG::2012-08-30 13:00:15,037::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1839::DEBUG::2012-08-30 13:00:15,037::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1839::DEBUG::2012-08-30 13:00:15,037::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1839::DEBUG::2012-08-30 13:00:15,037::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1839::DEBUG::2012-08-30 13:00:15,037::task::978::TaskManager.Task::(_decref) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:00:15,340::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:15,340::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:18,307::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:20,341::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:23,308::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:23,309::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1841::DEBUG::2012-08-30 13:00:24,975::task::588::TaskManager.Task::(_updateState) Task=`5d2be228-e3a4-4d30-bfe1-eca937fa1a77`::moving from state init -> state preparing >Thread-1841::INFO::2012-08-30 13:00:24,975::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1841::INFO::2012-08-30 13:00:24,975::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161981582642', 'lastCheck': 1346302820.101984, 'code': 0, 'valid': True}} >Thread-1841::DEBUG::2012-08-30 13:00:24,975::task::1172::TaskManager.Task::(prepare) Task=`5d2be228-e3a4-4d30-bfe1-eca937fa1a77`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161981582642', 'lastCheck': 1346302820.101984, 'code': 0, 'valid': True}} >Thread-1841::DEBUG::2012-08-30 13:00:24,976::task::588::TaskManager.Task::(_updateState) Task=`5d2be228-e3a4-4d30-bfe1-eca937fa1a77`::moving from state preparing -> state finished >Thread-1841::DEBUG::2012-08-30 13:00:24,976::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1841::DEBUG::2012-08-30 13:00:24,976::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1841::DEBUG::2012-08-30 13:00:24,976::task::978::TaskManager.Task::(_decref) Task=`5d2be228-e3a4-4d30-bfe1-eca937fa1a77`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:00:25,342::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:00:25,540::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:00:28,310::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-26::DEBUG::2012-08-30 13:00:30,102::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:00:30,102::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-26::DEBUG::2012-08-30 13:00:30,103::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None) >Thread-26::DEBUG::2012-08-30 13:00:30,213::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5 >Thread-26::WARNING::2012-08-30 13:00:30,214::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found'] >Thread-26::DEBUG::2012-08-30 13:00:30,214::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-26::DEBUG::2012-08-30 13:00:30,220::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:00:30,221::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend >Thread-26::DEBUG::2012-08-30 13:00:30,223::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d'] >Thread-26::DEBUG::2012-08-30 13:00:30,225::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-26::WARNING::2012-08-30 13:00:30,225::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered >Thread-26::WARNING::2012-08-30 13:00:30,225::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered >Thread-1842::DEBUG::2012-08-30 13:00:30,255::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1842::DEBUG::2012-08-30 13:00:30,256::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1843::DEBUG::2012-08-30 13:00:30,256::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1843::DEBUG::2012-08-30 13:00:30,257::task::588::TaskManager.Task::(_updateState) Task=`62bc07d9-3439-4390-a42b-4059a5ae9981`::moving from state init -> state preparing >Thread-1843::INFO::2012-08-30 13:00:30,258::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1843::INFO::2012-08-30 13:00:30,258::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1843::DEBUG::2012-08-30 13:00:30,259::task::1172::TaskManager.Task::(prepare) Task=`62bc07d9-3439-4390-a42b-4059a5ae9981`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1843::DEBUG::2012-08-30 13:00:30,259::task::588::TaskManager.Task::(_updateState) Task=`62bc07d9-3439-4390-a42b-4059a5ae9981`::moving from state preparing -> state finished >Thread-1843::DEBUG::2012-08-30 13:00:30,259::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1843::DEBUG::2012-08-30 13:00:30,259::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1843::DEBUG::2012-08-30 13:00:30,260::task::978::TaskManager.Task::(_decref) Task=`62bc07d9-3439-4390-a42b-4059a5ae9981`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 13:00:30,343::task::588::TaskManager.Task::(_updateState) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:00:30,343::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:00:30,344::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a7e14fd6-a963-4831-9a04-ce61465bb232`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:00:30,344::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:00:30,344::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:00:30,345::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a7e14fd6-a963-4831-9a04-ce61465bb232`::Granted request >Thread-104::DEBUG::2012-08-30 13:00:30,345::task::817::TaskManager.Task::(resourceAcquired) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:00:30,345::task::978::TaskManager.Task::(_decref) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:00:30,347::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:00:30,349::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:00:30,350::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:00:30,350::task::1172::TaskManager.Task::(prepare) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:00:30,350::task::588::TaskManager.Task::(_updateState) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:00:30,351::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:00:30,351::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:00:30,351::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:00:30,351::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:00:30,352::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:00:30,352::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:00:30,352::task::978::TaskManager.Task::(_decref) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:00:30,352::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:30,353::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:30,353::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:30,354::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:00:32,546::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:00:33,311::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1849::DEBUG::2012-08-30 13:00:33,668::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1849::DEBUG::2012-08-30 13:00:33,669::task::588::TaskManager.Task::(_updateState) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::moving from state init -> state preparing >Thread-1849::INFO::2012-08-30 13:00:33,669::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1849::DEBUG::2012-08-30 13:00:33,669::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`aa45b25a-b941-4f85-b7ab-5334500e9cf8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1849::DEBUG::2012-08-30 13:00:33,670::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1849::DEBUG::2012-08-30 13:00:33,670::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1849::DEBUG::2012-08-30 13:00:33,670::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`aa45b25a-b941-4f85-b7ab-5334500e9cf8`::Granted request >Thread-1849::DEBUG::2012-08-30 13:00:33,670::task::817::TaskManager.Task::(resourceAcquired) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1849::DEBUG::2012-08-30 13:00:33,671::task::978::TaskManager.Task::(_decref) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::ref 1 aborting False >Thread-1849::INFO::2012-08-30 13:00:33,674::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1849::DEBUG::2012-08-30 13:00:33,674::task::1172::TaskManager.Task::(prepare) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1849::DEBUG::2012-08-30 13:00:33,674::task::588::TaskManager.Task::(_updateState) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::moving from state preparing -> state finished >Thread-1849::DEBUG::2012-08-30 13:00:33,674::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1849::DEBUG::2012-08-30 13:00:33,675::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1849::DEBUG::2012-08-30 13:00:33,675::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1849::DEBUG::2012-08-30 13:00:33,675::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1849::DEBUG::2012-08-30 13:00:33,675::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1849::DEBUG::2012-08-30 13:00:33,676::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1849::DEBUG::2012-08-30 13:00:33,676::task::978::TaskManager.Task::(_decref) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:00:35,355::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:38,313::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:38,313::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:40,356::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:43,314::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:45,357::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:45,357::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1853::DEBUG::2012-08-30 13:00:46,915::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1853::DEBUG::2012-08-30 13:00:46,915::task::588::TaskManager.Task::(_updateState) Task=`b2e3615e-59a6-4398-a9fc-a2069e22f591`::moving from state init -> state preparing >Thread-1853::INFO::2012-08-30 13:00:46,915::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1853::INFO::2012-08-30 13:00:46,916::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1853::DEBUG::2012-08-30 13:00:46,916::task::1172::TaskManager.Task::(prepare) Task=`b2e3615e-59a6-4398-a9fc-a2069e22f591`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1853::DEBUG::2012-08-30 13:00:46,916::task::588::TaskManager.Task::(_updateState) Task=`b2e3615e-59a6-4398-a9fc-a2069e22f591`::moving from state preparing -> state finished >Thread-1853::DEBUG::2012-08-30 13:00:46,916::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1853::DEBUG::2012-08-30 13:00:46,916::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1853::DEBUG::2012-08-30 13:00:46,917::task::978::TaskManager.Task::(_decref) Task=`b2e3615e-59a6-4398-a9fc-a2069e22f591`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:00:48,316::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1854::DEBUG::2012-08-30 13:00:50,117::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1854::DEBUG::2012-08-30 13:00:50,117::task::588::TaskManager.Task::(_updateState) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::moving from state init -> state preparing >Thread-1854::INFO::2012-08-30 13:00:50,118::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1854::DEBUG::2012-08-30 13:00:50,118::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`56543bce-781a-4421-b8b3-014c80183452`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1854::DEBUG::2012-08-30 13:00:50,118::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1854::DEBUG::2012-08-30 13:00:50,119::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1854::DEBUG::2012-08-30 13:00:50,119::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`56543bce-781a-4421-b8b3-014c80183452`::Granted request >Thread-1854::DEBUG::2012-08-30 13:00:50,119::task::817::TaskManager.Task::(resourceAcquired) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1854::DEBUG::2012-08-30 13:00:50,119::task::978::TaskManager.Task::(_decref) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::ref 1 aborting False >Thread-1854::INFO::2012-08-30 13:00:50,122::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1854::DEBUG::2012-08-30 13:00:50,123::task::1172::TaskManager.Task::(prepare) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1854::DEBUG::2012-08-30 13:00:50,123::task::588::TaskManager.Task::(_updateState) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::moving from state preparing -> state finished >Thread-1854::DEBUG::2012-08-30 13:00:50,123::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1854::DEBUG::2012-08-30 13:00:50,123::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1854::DEBUG::2012-08-30 13:00:50,124::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1854::DEBUG::2012-08-30 13:00:50,124::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1854::DEBUG::2012-08-30 13:00:50,124::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1854::DEBUG::2012-08-30 13:00:50,124::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1854::DEBUG::2012-08-30 13:00:50,125::task::978::TaskManager.Task::(_decref) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:00:50,358::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:00:53,316::task::588::TaskManager.Task::(_updateState) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:00:53,317::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:00:53,317::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d223f689-ef3d-4e0e-b121-b5d5f8451a96`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:00:53,318::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:00:53,318::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:00:53,318::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d223f689-ef3d-4e0e-b121-b5d5f8451a96`::Granted request >Thread-40::DEBUG::2012-08-30 13:00:53,318::task::817::TaskManager.Task::(resourceAcquired) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:00:53,319::task::978::TaskManager.Task::(_decref) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:00:53,320::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:00:53,322::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:00:53,324::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:00:53,324::task::1172::TaskManager.Task::(prepare) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:00:53,324::task::588::TaskManager.Task::(_updateState) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:00:53,324::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:00:53,324::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:00:53,325::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:00:53,325::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:00:53,325::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:00:53,325::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:00:53,326::task::978::TaskManager.Task::(_decref) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:00:53,326::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:53,326::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:53,327::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:00:53,327::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:00:55,359::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1856::DEBUG::2012-08-30 13:00:55,368::task::588::TaskManager.Task::(_updateState) Task=`18ad438a-9f67-4869-ab46-6dc621bac2e0`::moving from state init -> state preparing >Thread-1856::INFO::2012-08-30 13:00:55,368::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1856::INFO::2012-08-30 13:00:55,368::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177001953125', 'lastCheck': 1346302850.244632, 'code': 0, 'valid': True}} >Thread-1856::DEBUG::2012-08-30 13:00:55,368::task::1172::TaskManager.Task::(prepare) Task=`18ad438a-9f67-4869-ab46-6dc621bac2e0`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177001953125', 'lastCheck': 1346302850.244632, 'code': 0, 'valid': True}} >Thread-1856::DEBUG::2012-08-30 13:00:55,369::task::588::TaskManager.Task::(_updateState) Task=`18ad438a-9f67-4869-ab46-6dc621bac2e0`::moving from state preparing -> state finished >Thread-1856::DEBUG::2012-08-30 13:00:55,369::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1856::DEBUG::2012-08-30 13:00:55,369::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1856::DEBUG::2012-08-30 13:00:55,369::task::978::TaskManager.Task::(_decref) Task=`18ad438a-9f67-4869-ab46-6dc621bac2e0`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:00:55,567::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:00:58,328::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:00,360::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:00,360::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1857::DEBUG::2012-08-30 13:01:00,710::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1857::DEBUG::2012-08-30 13:01:00,711::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >VM Channels Listener::DEBUG::2012-08-30 13:01:02,571::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:01:03,330::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1858::DEBUG::2012-08-30 13:01:03,524::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1858::DEBUG::2012-08-30 13:01:03,524::task::588::TaskManager.Task::(_updateState) Task=`68cc0cc0-92e7-40b6-850d-aa4bef7a3a33`::moving from state init -> state preparing >Thread-1858::INFO::2012-08-30 13:01:03,525::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1858::INFO::2012-08-30 13:01:03,525::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1858::DEBUG::2012-08-30 13:01:03,525::task::1172::TaskManager.Task::(prepare) Task=`68cc0cc0-92e7-40b6-850d-aa4bef7a3a33`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1858::DEBUG::2012-08-30 13:01:03,525::task::588::TaskManager.Task::(_updateState) Task=`68cc0cc0-92e7-40b6-850d-aa4bef7a3a33`::moving from state preparing -> state finished >Thread-1858::DEBUG::2012-08-30 13:01:03,526::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1858::DEBUG::2012-08-30 13:01:03,526::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1858::DEBUG::2012-08-30 13:01:03,526::task::978::TaskManager.Task::(_decref) Task=`68cc0cc0-92e7-40b6-850d-aa4bef7a3a33`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:01:05,361::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1860::DEBUG::2012-08-30 13:01:06,778::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1860::DEBUG::2012-08-30 13:01:06,779::task::588::TaskManager.Task::(_updateState) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::moving from state init -> state preparing >Thread-1860::INFO::2012-08-30 13:01:06,779::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1860::DEBUG::2012-08-30 13:01:06,779::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6913cea6-c059-4842-9c9d-b59fea28eae7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1860::DEBUG::2012-08-30 13:01:06,780::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1860::DEBUG::2012-08-30 13:01:06,780::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1860::DEBUG::2012-08-30 13:01:06,780::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6913cea6-c059-4842-9c9d-b59fea28eae7`::Granted request >Thread-1860::DEBUG::2012-08-30 13:01:06,780::task::817::TaskManager.Task::(resourceAcquired) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1860::DEBUG::2012-08-30 13:01:06,781::task::978::TaskManager.Task::(_decref) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::ref 1 aborting False >Thread-1860::INFO::2012-08-30 13:01:06,784::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1860::DEBUG::2012-08-30 13:01:06,784::task::1172::TaskManager.Task::(prepare) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1860::DEBUG::2012-08-30 13:01:06,784::task::588::TaskManager.Task::(_updateState) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::moving from state preparing -> state finished >Thread-1860::DEBUG::2012-08-30 13:01:06,784::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1860::DEBUG::2012-08-30 13:01:06,784::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1860::DEBUG::2012-08-30 13:01:06,785::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1860::DEBUG::2012-08-30 13:01:06,785::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1860::DEBUG::2012-08-30 13:01:06,785::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1860::DEBUG::2012-08-30 13:01:06,785::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1860::DEBUG::2012-08-30 13:01:06,786::task::978::TaskManager.Task::(_decref) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:01:08,331::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:08,332::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:10,362::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:13,333::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:15,363::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:15,364::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:18,334::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1863::DEBUG::2012-08-30 13:01:20,031::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1863::DEBUG::2012-08-30 13:01:20,032::task::588::TaskManager.Task::(_updateState) Task=`b5460281-83ee-44c3-aff4-af80d9e4f6b9`::moving from state init -> state preparing >Thread-1863::INFO::2012-08-30 13:01:20,032::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1863::INFO::2012-08-30 13:01:20,032::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1863::DEBUG::2012-08-30 13:01:20,032::task::1172::TaskManager.Task::(prepare) Task=`b5460281-83ee-44c3-aff4-af80d9e4f6b9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1863::DEBUG::2012-08-30 13:01:20,033::task::588::TaskManager.Task::(_updateState) Task=`b5460281-83ee-44c3-aff4-af80d9e4f6b9`::moving from state preparing -> state finished >Thread-1863::DEBUG::2012-08-30 13:01:20,033::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1863::DEBUG::2012-08-30 13:01:20,033::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1863::DEBUG::2012-08-30 13:01:20,033::task::978::TaskManager.Task::(_decref) Task=`b5460281-83ee-44c3-aff4-af80d9e4f6b9`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:01:20,365::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:23,336::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:23,336::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1865::DEBUG::2012-08-30 13:01:25,129::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1865::DEBUG::2012-08-30 13:01:25,129::task::588::TaskManager.Task::(_updateState) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::moving from state init -> state preparing >Thread-1865::INFO::2012-08-30 13:01:25,129::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1865::DEBUG::2012-08-30 13:01:25,130::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`dd01ec3d-337f-4269-b847-7784c7747b8f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1865::DEBUG::2012-08-30 13:01:25,130::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1865::DEBUG::2012-08-30 13:01:25,130::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1865::DEBUG::2012-08-30 13:01:25,130::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`dd01ec3d-337f-4269-b847-7784c7747b8f`::Granted request >Thread-1865::DEBUG::2012-08-30 13:01:25,131::task::817::TaskManager.Task::(resourceAcquired) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1865::DEBUG::2012-08-30 13:01:25,131::task::978::TaskManager.Task::(_decref) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::ref 1 aborting False >Thread-1865::INFO::2012-08-30 13:01:25,134::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1865::DEBUG::2012-08-30 13:01:25,134::task::1172::TaskManager.Task::(prepare) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1865::DEBUG::2012-08-30 13:01:25,134::task::588::TaskManager.Task::(_updateState) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::moving from state preparing -> state finished >Thread-1865::DEBUG::2012-08-30 13:01:25,135::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1865::DEBUG::2012-08-30 13:01:25,135::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1865::DEBUG::2012-08-30 13:01:25,135::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1865::DEBUG::2012-08-30 13:01:25,135::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1865::DEBUG::2012-08-30 13:01:25,136::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1865::DEBUG::2012-08-30 13:01:25,136::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1865::DEBUG::2012-08-30 13:01:25,136::task::978::TaskManager.Task::(_decref) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:01:25,365::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:01:25,590::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1866::DEBUG::2012-08-30 13:01:26,768::task::588::TaskManager.Task::(_updateState) Task=`b4ef5710-9499-46bc-9c16-fb382445a835`::moving from state init -> state preparing >Thread-1866::INFO::2012-08-30 13:01:26,768::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1866::INFO::2012-08-30 13:01:26,768::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162816047668', 'lastCheck': 1346302880.2646329, 'code': 0, 'valid': True}} >Thread-1866::DEBUG::2012-08-30 13:01:26,768::task::1172::TaskManager.Task::(prepare) Task=`b4ef5710-9499-46bc-9c16-fb382445a835`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162816047668', 'lastCheck': 1346302880.2646329, 'code': 0, 'valid': True}} >Thread-1866::DEBUG::2012-08-30 13:01:26,769::task::588::TaskManager.Task::(_updateState) Task=`b4ef5710-9499-46bc-9c16-fb382445a835`::moving from state preparing -> state finished >Thread-1866::DEBUG::2012-08-30 13:01:26,769::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1866::DEBUG::2012-08-30 13:01:26,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1866::DEBUG::2012-08-30 13:01:26,769::task::978::TaskManager.Task::(_decref) Task=`b4ef5710-9499-46bc-9c16-fb382445a835`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:01:28,337::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1867::DEBUG::2012-08-30 13:01:30,082::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1867::DEBUG::2012-08-30 13:01:30,082::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::DEBUG::2012-08-30 13:01:30,366::task::588::TaskManager.Task::(_updateState) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:01:30,367::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:01:30,367::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`eef7b159-3f07-42b7-905a-0937a3379e5b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:01:30,367::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:01:30,368::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:01:30,368::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`eef7b159-3f07-42b7-905a-0937a3379e5b`::Granted request >Thread-104::DEBUG::2012-08-30 13:01:30,368::task::817::TaskManager.Task::(resourceAcquired) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:01:30,368::task::978::TaskManager.Task::(_decref) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:01:30,370::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:01:30,372::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:01:30,373::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:01:30,373::task::1172::TaskManager.Task::(prepare) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:01:30,373::task::588::TaskManager.Task::(_updateState) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:01:30,374::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:01:30,374::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:01:30,374::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:01:30,374::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:01:30,375::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:01:30,375::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:01:30,375::task::978::TaskManager.Task::(_decref) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:01:30,375::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:30,376::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:30,376::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:30,377::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:01:32,593::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:01:33,339::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:35,378::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:38,340::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:38,340::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1869::DEBUG::2012-08-30 13:01:38,427::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1869::DEBUG::2012-08-30 13:01:38,427::task::588::TaskManager.Task::(_updateState) Task=`9705d26c-d7bb-4095-975c-4c356c4c063e`::moving from state init -> state preparing >Thread-1869::INFO::2012-08-30 13:01:38,428::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1869::INFO::2012-08-30 13:01:38,428::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1869::DEBUG::2012-08-30 13:01:38,428::task::1172::TaskManager.Task::(prepare) Task=`9705d26c-d7bb-4095-975c-4c356c4c063e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1869::DEBUG::2012-08-30 13:01:38,428::task::588::TaskManager.Task::(_updateState) Task=`9705d26c-d7bb-4095-975c-4c356c4c063e`::moving from state preparing -> state finished >Thread-1869::DEBUG::2012-08-30 13:01:38,428::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1869::DEBUG::2012-08-30 13:01:38,429::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1869::DEBUG::2012-08-30 13:01:38,429::task::978::TaskManager.Task::(_decref) Task=`9705d26c-d7bb-4095-975c-4c356c4c063e`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:01:40,379::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1871::DEBUG::2012-08-30 13:01:41,795::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1871::DEBUG::2012-08-30 13:01:41,795::task::588::TaskManager.Task::(_updateState) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::moving from state init -> state preparing >Thread-1871::INFO::2012-08-30 13:01:41,796::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1871::DEBUG::2012-08-30 13:01:41,796::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`782091b4-c9ab-4175-a65b-1a66ba270615`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1871::DEBUG::2012-08-30 13:01:41,796::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1871::DEBUG::2012-08-30 13:01:41,796::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1871::DEBUG::2012-08-30 13:01:41,797::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`782091b4-c9ab-4175-a65b-1a66ba270615`::Granted request >Thread-1871::DEBUG::2012-08-30 13:01:41,797::task::817::TaskManager.Task::(resourceAcquired) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1871::DEBUG::2012-08-30 13:01:41,797::task::978::TaskManager.Task::(_decref) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::ref 1 aborting False >Thread-1871::INFO::2012-08-30 13:01:41,800::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1871::DEBUG::2012-08-30 13:01:41,800::task::1172::TaskManager.Task::(prepare) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1871::DEBUG::2012-08-30 13:01:41,801::task::588::TaskManager.Task::(_updateState) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::moving from state preparing -> state finished >Thread-1871::DEBUG::2012-08-30 13:01:41,801::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1871::DEBUG::2012-08-30 13:01:41,801::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1871::DEBUG::2012-08-30 13:01:41,801::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1871::DEBUG::2012-08-30 13:01:41,802::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1871::DEBUG::2012-08-30 13:01:41,802::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1871::DEBUG::2012-08-30 13:01:41,802::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1871::DEBUG::2012-08-30 13:01:41,802::task::978::TaskManager.Task::(_decref) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:01:43,341::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:45,380::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:45,380::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:48,343::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:01:50,381::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:01:53,344::task::588::TaskManager.Task::(_updateState) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:01:53,344::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:01:53,345::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c61d42a6-ba38-453a-be83-06a22ca69600`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:01:53,345::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:01:53,345::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:01:53,345::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c61d42a6-ba38-453a-be83-06a22ca69600`::Granted request >Thread-40::DEBUG::2012-08-30 13:01:53,346::task::817::TaskManager.Task::(resourceAcquired) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:01:53,346::task::978::TaskManager.Task::(_decref) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:01:53,348::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:01:53,350::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:01:53,351::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:01:53,351::task::1172::TaskManager.Task::(prepare) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:01:53,351::task::588::TaskManager.Task::(_updateState) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:01:53,352::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:01:53,352::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:01:53,352::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:01:53,352::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:01:53,353::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:01:53,353::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:01:53,353::task::978::TaskManager.Task::(_decref) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:01:53,354::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:53,354::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:53,354::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:01:53,355::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1874::DEBUG::2012-08-30 13:01:55,125::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1874::DEBUG::2012-08-30 13:01:55,125::task::588::TaskManager.Task::(_updateState) Task=`e3ddc274-97ff-48a6-8808-cee8f2a115b8`::moving from state init -> state preparing >Thread-1874::INFO::2012-08-30 13:01:55,125::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1874::INFO::2012-08-30 13:01:55,126::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1874::DEBUG::2012-08-30 13:01:55,126::task::1172::TaskManager.Task::(prepare) Task=`e3ddc274-97ff-48a6-8808-cee8f2a115b8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1874::DEBUG::2012-08-30 13:01:55,126::task::588::TaskManager.Task::(_updateState) Task=`e3ddc274-97ff-48a6-8808-cee8f2a115b8`::moving from state preparing -> state finished >Thread-1874::DEBUG::2012-08-30 13:01:55,126::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1874::DEBUG::2012-08-30 13:01:55,126::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1874::DEBUG::2012-08-30 13:01:55,127::task::978::TaskManager.Task::(_decref) Task=`e3ddc274-97ff-48a6-8808-cee8f2a115b8`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:01:55,382::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:01:55,615::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1875::DEBUG::2012-08-30 13:01:55,932::task::588::TaskManager.Task::(_updateState) Task=`884daea6-e683-422a-bf59-dea93fe65d0b`::moving from state init -> state preparing >Thread-1875::INFO::2012-08-30 13:01:55,932::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1875::INFO::2012-08-30 13:01:55,933::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176906585693', 'lastCheck': 1346302910.284256, 'code': 0, 'valid': True}} >Thread-1875::DEBUG::2012-08-30 13:01:55,933::task::1172::TaskManager.Task::(prepare) Task=`884daea6-e683-422a-bf59-dea93fe65d0b`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176906585693', 'lastCheck': 1346302910.284256, 'code': 0, 'valid': True}} >Thread-1875::DEBUG::2012-08-30 13:01:55,933::task::588::TaskManager.Task::(_updateState) Task=`884daea6-e683-422a-bf59-dea93fe65d0b`::moving from state preparing -> state finished >Thread-1875::DEBUG::2012-08-30 13:01:55,933::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1875::DEBUG::2012-08-30 13:01:55,934::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1875::DEBUG::2012-08-30 13:01:55,934::task::978::TaskManager.Task::(_decref) Task=`884daea6-e683-422a-bf59-dea93fe65d0b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:01:58,356::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1876::DEBUG::2012-08-30 13:01:58,373::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1876::DEBUG::2012-08-30 13:01:58,373::task::588::TaskManager.Task::(_updateState) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::moving from state init -> state preparing >Thread-1876::INFO::2012-08-30 13:01:58,374::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1876::DEBUG::2012-08-30 13:01:58,374::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e97fa825-53b2-4687-92ea-1130ec2325f7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1876::DEBUG::2012-08-30 13:01:58,374::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1876::DEBUG::2012-08-30 13:01:58,374::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1876::DEBUG::2012-08-30 13:01:58,375::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e97fa825-53b2-4687-92ea-1130ec2325f7`::Granted request >Thread-1876::DEBUG::2012-08-30 13:01:58,375::task::817::TaskManager.Task::(resourceAcquired) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1876::DEBUG::2012-08-30 13:01:58,375::task::978::TaskManager.Task::(_decref) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::ref 1 aborting False >Thread-1876::INFO::2012-08-30 13:01:58,378::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1876::DEBUG::2012-08-30 13:01:58,378::task::1172::TaskManager.Task::(prepare) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1876::DEBUG::2012-08-30 13:01:58,379::task::588::TaskManager.Task::(_updateState) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::moving from state preparing -> state finished >Thread-1876::DEBUG::2012-08-30 13:01:58,379::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1876::DEBUG::2012-08-30 13:01:58,379::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1876::DEBUG::2012-08-30 13:01:58,379::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1876::DEBUG::2012-08-30 13:01:58,380::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1876::DEBUG::2012-08-30 13:01:58,380::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1876::DEBUG::2012-08-30 13:01:58,380::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1876::DEBUG::2012-08-30 13:01:58,380::task::978::TaskManager.Task::(_decref) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::ref 0 aborting False >Thread-1877::DEBUG::2012-08-30 13:01:59,216::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1877::DEBUG::2012-08-30 13:01:59,216::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:02:00,383::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:00,383::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:02:02,622::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:02:03,357::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:05,384::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:08,358::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:08,359::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:10,385::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1880::DEBUG::2012-08-30 13:02:11,309::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1880::DEBUG::2012-08-30 13:02:11,309::task::588::TaskManager.Task::(_updateState) Task=`20d392de-1b87-48cb-8299-08a2bc17f94a`::moving from state init -> state preparing >Thread-1880::INFO::2012-08-30 13:02:11,309::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1880::INFO::2012-08-30 13:02:11,310::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1880::DEBUG::2012-08-30 13:02:11,310::task::1172::TaskManager.Task::(prepare) Task=`20d392de-1b87-48cb-8299-08a2bc17f94a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1880::DEBUG::2012-08-30 13:02:11,310::task::588::TaskManager.Task::(_updateState) Task=`20d392de-1b87-48cb-8299-08a2bc17f94a`::moving from state preparing -> state finished >Thread-1880::DEBUG::2012-08-30 13:02:11,310::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1880::DEBUG::2012-08-30 13:02:11,310::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1880::DEBUG::2012-08-30 13:02:11,311::task::978::TaskManager.Task::(_decref) Task=`20d392de-1b87-48cb-8299-08a2bc17f94a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:02:13,360::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1881::DEBUG::2012-08-30 13:02:14,568::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1881::DEBUG::2012-08-30 13:02:14,569::task::588::TaskManager.Task::(_updateState) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::moving from state init -> state preparing >Thread-1881::INFO::2012-08-30 13:02:14,569::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1881::DEBUG::2012-08-30 13:02:14,570::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0f180446-afa3-4a8b-900f-89b9fb1cc0e2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1881::DEBUG::2012-08-30 13:02:14,570::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1881::DEBUG::2012-08-30 13:02:14,570::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1881::DEBUG::2012-08-30 13:02:14,571::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0f180446-afa3-4a8b-900f-89b9fb1cc0e2`::Granted request >Thread-1881::DEBUG::2012-08-30 13:02:14,571::task::817::TaskManager.Task::(resourceAcquired) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1881::DEBUG::2012-08-30 13:02:14,571::task::978::TaskManager.Task::(_decref) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::ref 1 aborting False >Thread-1881::INFO::2012-08-30 13:02:14,574::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1881::DEBUG::2012-08-30 13:02:14,574::task::1172::TaskManager.Task::(prepare) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1881::DEBUG::2012-08-30 13:02:14,575::task::588::TaskManager.Task::(_updateState) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::moving from state preparing -> state finished >Thread-1881::DEBUG::2012-08-30 13:02:14,575::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1881::DEBUG::2012-08-30 13:02:14,575::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1881::DEBUG::2012-08-30 13:02:14,575::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1881::DEBUG::2012-08-30 13:02:14,576::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1881::DEBUG::2012-08-30 13:02:14,576::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1881::DEBUG::2012-08-30 13:02:14,576::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1881::DEBUG::2012-08-30 13:02:14,576::task::978::TaskManager.Task::(_decref) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:02:15,386::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:15,387::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:18,361::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:20,388::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:23,363::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:23,363::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1884::DEBUG::2012-08-30 13:02:24,937::task::588::TaskManager.Task::(_updateState) Task=`e1e80c4a-3f45-4352-8c01-445f27f01ebf`::moving from state init -> state preparing >Thread-1884::INFO::2012-08-30 13:02:24,938::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1884::INFO::2012-08-30 13:02:24,938::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163197517395', 'lastCheck': 1346302940.304249, 'code': 0, 'valid': True}} >Thread-1884::DEBUG::2012-08-30 13:02:24,938::task::1172::TaskManager.Task::(prepare) Task=`e1e80c4a-3f45-4352-8c01-445f27f01ebf`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163197517395', 'lastCheck': 1346302940.304249, 'code': 0, 'valid': True}} >Thread-1884::DEBUG::2012-08-30 13:02:24,938::task::588::TaskManager.Task::(_updateState) Task=`e1e80c4a-3f45-4352-8c01-445f27f01ebf`::moving from state preparing -> state finished >Thread-1884::DEBUG::2012-08-30 13:02:24,939::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1884::DEBUG::2012-08-30 13:02:24,939::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1884::DEBUG::2012-08-30 13:02:24,939::task::978::TaskManager.Task::(_decref) Task=`e1e80c4a-3f45-4352-8c01-445f27f01ebf`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:02:25,389::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:02:25,640::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:02:28,364::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1885::DEBUG::2012-08-30 13:02:30,213::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1885::DEBUG::2012-08-30 13:02:30,213::task::588::TaskManager.Task::(_updateState) Task=`65a724e0-4406-4e8b-ad55-cc3f59e2cd6c`::moving from state init -> state preparing >Thread-1885::INFO::2012-08-30 13:02:30,213::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1885::INFO::2012-08-30 13:02:30,214::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1885::DEBUG::2012-08-30 13:02:30,214::task::1172::TaskManager.Task::(prepare) Task=`65a724e0-4406-4e8b-ad55-cc3f59e2cd6c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1885::DEBUG::2012-08-30 13:02:30,214::task::588::TaskManager.Task::(_updateState) Task=`65a724e0-4406-4e8b-ad55-cc3f59e2cd6c`::moving from state preparing -> state finished >Thread-1885::DEBUG::2012-08-30 13:02:30,214::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1885::DEBUG::2012-08-30 13:02:30,215::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1885::DEBUG::2012-08-30 13:02:30,215::task::978::TaskManager.Task::(_decref) Task=`65a724e0-4406-4e8b-ad55-cc3f59e2cd6c`::ref 0 aborting False >Thread-1886::DEBUG::2012-08-30 13:02:30,216::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1886::DEBUG::2012-08-30 13:02:30,216::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::DEBUG::2012-08-30 13:02:30,390::task::588::TaskManager.Task::(_updateState) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:02:30,390::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:02:30,390::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`43d5a160-00d7-41c2-89e3-3a25256f1b66`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:02:30,390::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:02:30,391::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:02:30,391::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`43d5a160-00d7-41c2-89e3-3a25256f1b66`::Granted request >Thread-104::DEBUG::2012-08-30 13:02:30,391::task::817::TaskManager.Task::(resourceAcquired) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:02:30,392::task::978::TaskManager.Task::(_decref) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:02:30,393::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:02:30,395::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:02:30,396::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:02:30,396::task::1172::TaskManager.Task::(prepare) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:02:30,397::task::588::TaskManager.Task::(_updateState) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:02:30,397::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:02:30,397::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:02:30,397::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:02:30,397::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:02:30,398::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:02:30,398::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:02:30,398::task::978::TaskManager.Task::(_decref) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:02:30,399::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:30,399::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:30,399::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:30,400::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:02:32,647::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:02:33,365::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1887::DEBUG::2012-08-30 13:02:33,404::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1887::DEBUG::2012-08-30 13:02:33,404::task::588::TaskManager.Task::(_updateState) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::moving from state init -> state preparing >Thread-1887::INFO::2012-08-30 13:02:33,404::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1887::DEBUG::2012-08-30 13:02:33,405::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`7ded09c5-e9d2-4c19-9f43-fc63494f4c91`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1887::DEBUG::2012-08-30 13:02:33,405::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1887::DEBUG::2012-08-30 13:02:33,405::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1887::DEBUG::2012-08-30 13:02:33,405::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`7ded09c5-e9d2-4c19-9f43-fc63494f4c91`::Granted request >Thread-1887::DEBUG::2012-08-30 13:02:33,406::task::817::TaskManager.Task::(resourceAcquired) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1887::DEBUG::2012-08-30 13:02:33,406::task::978::TaskManager.Task::(_decref) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::ref 1 aborting False >Thread-1887::INFO::2012-08-30 13:02:33,409::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1887::DEBUG::2012-08-30 13:02:33,409::task::1172::TaskManager.Task::(prepare) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1887::DEBUG::2012-08-30 13:02:33,409::task::588::TaskManager.Task::(_updateState) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::moving from state preparing -> state finished >Thread-1887::DEBUG::2012-08-30 13:02:33,410::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1887::DEBUG::2012-08-30 13:02:33,410::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1887::DEBUG::2012-08-30 13:02:33,410::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1887::DEBUG::2012-08-30 13:02:33,410::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1887::DEBUG::2012-08-30 13:02:33,411::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1887::DEBUG::2012-08-30 13:02:33,411::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1887::DEBUG::2012-08-30 13:02:33,411::task::978::TaskManager.Task::(_decref) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:02:35,401::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:38,366::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:38,367::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:40,402::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:43,368::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:45,403::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:45,403::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1891::DEBUG::2012-08-30 13:02:46,855::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1891::DEBUG::2012-08-30 13:02:46,856::task::588::TaskManager.Task::(_updateState) Task=`c0a52f9b-1634-438d-a3e2-5d970bb60bc8`::moving from state init -> state preparing >Thread-1891::INFO::2012-08-30 13:02:46,856::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1891::INFO::2012-08-30 13:02:46,856::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1891::DEBUG::2012-08-30 13:02:46,857::task::1172::TaskManager.Task::(prepare) Task=`c0a52f9b-1634-438d-a3e2-5d970bb60bc8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1891::DEBUG::2012-08-30 13:02:46,857::task::588::TaskManager.Task::(_updateState) Task=`c0a52f9b-1634-438d-a3e2-5d970bb60bc8`::moving from state preparing -> state finished >Thread-1891::DEBUG::2012-08-30 13:02:46,857::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1891::DEBUG::2012-08-30 13:02:46,857::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1891::DEBUG::2012-08-30 13:02:46,858::task::978::TaskManager.Task::(_decref) Task=`c0a52f9b-1634-438d-a3e2-5d970bb60bc8`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:02:48,369::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1892::DEBUG::2012-08-30 13:02:50,126::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1892::DEBUG::2012-08-30 13:02:50,126::task::588::TaskManager.Task::(_updateState) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::moving from state init -> state preparing >Thread-1892::INFO::2012-08-30 13:02:50,127::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1892::DEBUG::2012-08-30 13:02:50,128::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0a77e28f-efe7-4443-b664-02e4dcf76393`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1892::DEBUG::2012-08-30 13:02:50,128::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1892::DEBUG::2012-08-30 13:02:50,128::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1892::DEBUG::2012-08-30 13:02:50,128::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0a77e28f-efe7-4443-b664-02e4dcf76393`::Granted request >Thread-1892::DEBUG::2012-08-30 13:02:50,129::task::817::TaskManager.Task::(resourceAcquired) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1892::DEBUG::2012-08-30 13:02:50,129::task::978::TaskManager.Task::(_decref) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::ref 1 aborting False >Thread-1892::INFO::2012-08-30 13:02:50,132::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1892::DEBUG::2012-08-30 13:02:50,132::task::1172::TaskManager.Task::(prepare) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1892::DEBUG::2012-08-30 13:02:50,132::task::588::TaskManager.Task::(_updateState) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::moving from state preparing -> state finished >Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1892::DEBUG::2012-08-30 13:02:50,134::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1892::DEBUG::2012-08-30 13:02:50,134::task::978::TaskManager.Task::(_decref) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:02:50,404::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:02:53,370::task::588::TaskManager.Task::(_updateState) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:02:53,371::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:02:53,371::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`25fd5e2c-d7ec-40ce-bd34-4426fa57d707`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:02:53,372::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:02:53,372::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:02:53,372::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`25fd5e2c-d7ec-40ce-bd34-4426fa57d707`::Granted request >Thread-40::DEBUG::2012-08-30 13:02:53,372::task::817::TaskManager.Task::(resourceAcquired) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:02:53,373::task::978::TaskManager.Task::(_decref) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:02:53,374::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:02:53,377::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:02:53,378::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:02:53,378::task::1172::TaskManager.Task::(prepare) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:02:53,378::task::588::TaskManager.Task::(_updateState) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:02:53,378::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:02:53,380::task::978::TaskManager.Task::(_decref) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:02:53,380::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:53,381::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:53,381::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:02:53,381::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:02:55,405::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1894::DEBUG::2012-08-30 13:02:55,475::task::588::TaskManager.Task::(_updateState) Task=`a716f75a-0b35-41a3-a415-6e7855d10b18`::moving from state init -> state preparing >Thread-1894::INFO::2012-08-30 13:02:55,475::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1894::INFO::2012-08-30 13:02:55,475::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176095962524', 'lastCheck': 1346302970.323657, 'code': 0, 'valid': True}} >Thread-1894::DEBUG::2012-08-30 13:02:55,475::task::1172::TaskManager.Task::(prepare) Task=`a716f75a-0b35-41a3-a415-6e7855d10b18`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176095962524', 'lastCheck': 1346302970.323657, 'code': 0, 'valid': True}} >Thread-1894::DEBUG::2012-08-30 13:02:55,476::task::588::TaskManager.Task::(_updateState) Task=`a716f75a-0b35-41a3-a415-6e7855d10b18`::moving from state preparing -> state finished >Thread-1894::DEBUG::2012-08-30 13:02:55,476::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1894::DEBUG::2012-08-30 13:02:55,476::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1894::DEBUG::2012-08-30 13:02:55,476::task::978::TaskManager.Task::(_decref) Task=`a716f75a-0b35-41a3-a415-6e7855d10b18`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:02:55,669::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:02:58,382::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1895::DEBUG::2012-08-30 13:02:58,648::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1895::DEBUG::2012-08-30 13:02:58,648::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:03:00,406::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:00,406::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:03:02,676::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-1896::DEBUG::2012-08-30 13:03:03,323::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1896::DEBUG::2012-08-30 13:03:03,324::task::588::TaskManager.Task::(_updateState) Task=`a05ad071-a642-437c-a226-21e9b3d91b55`::moving from state init -> state preparing >Thread-1896::INFO::2012-08-30 13:03:03,324::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1896::INFO::2012-08-30 13:03:03,325::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1896::DEBUG::2012-08-30 13:03:03,325::task::1172::TaskManager.Task::(prepare) Task=`a05ad071-a642-437c-a226-21e9b3d91b55`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1896::DEBUG::2012-08-30 13:03:03,325::task::588::TaskManager.Task::(_updateState) Task=`a05ad071-a642-437c-a226-21e9b3d91b55`::moving from state preparing -> state finished >Thread-1896::DEBUG::2012-08-30 13:03:03,325::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1896::DEBUG::2012-08-30 13:03:03,325::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1896::DEBUG::2012-08-30 13:03:03,326::task::978::TaskManager.Task::(_decref) Task=`a05ad071-a642-437c-a226-21e9b3d91b55`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:03:03,383::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:05,407::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1898::DEBUG::2012-08-30 13:03:06,536::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1898::DEBUG::2012-08-30 13:03:06,537::task::588::TaskManager.Task::(_updateState) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::moving from state init -> state preparing >Thread-1898::INFO::2012-08-30 13:03:06,537::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1898::DEBUG::2012-08-30 13:03:06,537::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2dc91137-d70a-409c-a94b-ce10f0857207`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1898::DEBUG::2012-08-30 13:03:06,538::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1898::DEBUG::2012-08-30 13:03:06,538::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1898::DEBUG::2012-08-30 13:03:06,538::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2dc91137-d70a-409c-a94b-ce10f0857207`::Granted request >Thread-1898::DEBUG::2012-08-30 13:03:06,538::task::817::TaskManager.Task::(resourceAcquired) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1898::DEBUG::2012-08-30 13:03:06,539::task::978::TaskManager.Task::(_decref) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::ref 1 aborting False >Thread-1898::INFO::2012-08-30 13:03:06,542::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1898::DEBUG::2012-08-30 13:03:06,542::task::1172::TaskManager.Task::(prepare) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1898::DEBUG::2012-08-30 13:03:06,542::task::588::TaskManager.Task::(_updateState) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::moving from state preparing -> state finished >Thread-1898::DEBUG::2012-08-30 13:03:06,542::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1898::DEBUG::2012-08-30 13:03:06,544::task::978::TaskManager.Task::(_decref) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:03:08,385::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:08,385::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:10,408::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:13,386::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:15,409::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:15,410::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:18,387::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:20,410::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1902::DEBUG::2012-08-30 13:03:21,817::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1902::DEBUG::2012-08-30 13:03:21,817::task::588::TaskManager.Task::(_updateState) Task=`c83d8300-7370-4725-8681-e580af4687a6`::moving from state init -> state preparing >Thread-1902::INFO::2012-08-30 13:03:21,818::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1902::INFO::2012-08-30 13:03:21,818::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1902::DEBUG::2012-08-30 13:03:21,818::task::1172::TaskManager.Task::(prepare) Task=`c83d8300-7370-4725-8681-e580af4687a6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1902::DEBUG::2012-08-30 13:03:21,818::task::588::TaskManager.Task::(_updateState) Task=`c83d8300-7370-4725-8681-e580af4687a6`::moving from state preparing -> state finished >Thread-1902::DEBUG::2012-08-30 13:03:21,819::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1902::DEBUG::2012-08-30 13:03:21,819::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1902::DEBUG::2012-08-30 13:03:21,819::task::978::TaskManager.Task::(_decref) Task=`c83d8300-7370-4725-8681-e580af4687a6`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:03:23,389::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:23,390::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1903::DEBUG::2012-08-30 13:03:24,996::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1903::DEBUG::2012-08-30 13:03:24,996::task::588::TaskManager.Task::(_updateState) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::moving from state init -> state preparing >Thread-1903::INFO::2012-08-30 13:03:24,996::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1903::DEBUG::2012-08-30 13:03:24,997::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`47b2d6b4-0eb6-415a-ba0c-68a53635ecbc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1903::DEBUG::2012-08-30 13:03:24,997::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1903::DEBUG::2012-08-30 13:03:24,997::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1903::DEBUG::2012-08-30 13:03:24,997::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`47b2d6b4-0eb6-415a-ba0c-68a53635ecbc`::Granted request >Thread-1903::DEBUG::2012-08-30 13:03:24,998::task::817::TaskManager.Task::(resourceAcquired) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1903::DEBUG::2012-08-30 13:03:24,998::task::978::TaskManager.Task::(_decref) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::ref 1 aborting False >Thread-1903::INFO::2012-08-30 13:03:25,001::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1903::DEBUG::2012-08-30 13:03:25,001::task::1172::TaskManager.Task::(prepare) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1903::DEBUG::2012-08-30 13:03:25,001::task::588::TaskManager.Task::(_updateState) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::moving from state preparing -> state finished >Thread-1903::DEBUG::2012-08-30 13:03:25,002::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1903::DEBUG::2012-08-30 13:03:25,002::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1903::DEBUG::2012-08-30 13:03:25,002::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1903::DEBUG::2012-08-30 13:03:25,002::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1903::DEBUG::2012-08-30 13:03:25,003::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1903::DEBUG::2012-08-30 13:03:25,003::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1903::DEBUG::2012-08-30 13:03:25,003::task::978::TaskManager.Task::(_decref) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:03:25,411::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:03:25,697::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1904::DEBUG::2012-08-30 13:03:25,737::task::588::TaskManager.Task::(_updateState) Task=`a1142226-e541-4680-b82a-db4a8c56e568`::moving from state init -> state preparing >Thread-1904::INFO::2012-08-30 13:03:25,737::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1904::INFO::2012-08-30 13:03:25,738::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162386894226', 'lastCheck': 1346303000.3436971, 'code': 0, 'valid': True}} >Thread-1904::DEBUG::2012-08-30 13:03:25,738::task::1172::TaskManager.Task::(prepare) Task=`a1142226-e541-4680-b82a-db4a8c56e568`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162386894226', 'lastCheck': 1346303000.3436971, 'code': 0, 'valid': True}} >Thread-1904::DEBUG::2012-08-30 13:03:25,738::task::588::TaskManager.Task::(_updateState) Task=`a1142226-e541-4680-b82a-db4a8c56e568`::moving from state preparing -> state finished >Thread-1904::DEBUG::2012-08-30 13:03:25,738::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1904::DEBUG::2012-08-30 13:03:25,738::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1904::DEBUG::2012-08-30 13:03:25,739::task::978::TaskManager.Task::(_decref) Task=`a1142226-e541-4680-b82a-db4a8c56e568`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:03:28,391::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1905::DEBUG::2012-08-30 13:03:29,160::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1905::DEBUG::2012-08-30 13:03:29,160::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::DEBUG::2012-08-30 13:03:30,412::task::588::TaskManager.Task::(_updateState) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:03:30,413::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:03:30,413::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f7f46e10-6291-4d32-9f73-7a31310e7956`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:03:30,413::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:03:30,414::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:03:30,414::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f7f46e10-6291-4d32-9f73-7a31310e7956`::Granted request >Thread-104::DEBUG::2012-08-30 13:03:30,414::task::817::TaskManager.Task::(resourceAcquired) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:03:30,414::task::978::TaskManager.Task::(_decref) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:03:30,416::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:03:30,418::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:03:30,419::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:03:30,419::task::1172::TaskManager.Task::(prepare) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:03:30,419::task::588::TaskManager.Task::(_updateState) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:03:30,420::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:03:30,420::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:03:30,420::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:03:30,420::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:03:30,421::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:03:30,421::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:03:30,421::task::978::TaskManager.Task::(_decref) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:03:30,421::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:30,422::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:30,422::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:30,423::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:03:32,703::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:03:33,392::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:35,424::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1907::DEBUG::2012-08-30 13:03:37,971::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1907::DEBUG::2012-08-30 13:03:37,971::task::588::TaskManager.Task::(_updateState) Task=`9a9bdc34-93d4-4fc8-8f61-5b360e850d85`::moving from state init -> state preparing >Thread-1907::INFO::2012-08-30 13:03:37,971::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1907::INFO::2012-08-30 13:03:37,972::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1907::DEBUG::2012-08-30 13:03:37,972::task::1172::TaskManager.Task::(prepare) Task=`9a9bdc34-93d4-4fc8-8f61-5b360e850d85`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1907::DEBUG::2012-08-30 13:03:37,972::task::588::TaskManager.Task::(_updateState) Task=`9a9bdc34-93d4-4fc8-8f61-5b360e850d85`::moving from state preparing -> state finished >Thread-1907::DEBUG::2012-08-30 13:03:37,972::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1907::DEBUG::2012-08-30 13:03:37,972::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1907::DEBUG::2012-08-30 13:03:37,973::task::978::TaskManager.Task::(_decref) Task=`9a9bdc34-93d4-4fc8-8f61-5b360e850d85`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:03:38,393::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:38,394::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:40,425::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1909::DEBUG::2012-08-30 13:03:41,102::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1909::DEBUG::2012-08-30 13:03:41,102::task::588::TaskManager.Task::(_updateState) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::moving from state init -> state preparing >Thread-1909::INFO::2012-08-30 13:03:41,102::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1909::DEBUG::2012-08-30 13:03:41,103::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3807b644-377c-4da5-a3ff-07d16c0972d2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1909::DEBUG::2012-08-30 13:03:41,103::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1909::DEBUG::2012-08-30 13:03:41,103::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1909::DEBUG::2012-08-30 13:03:41,103::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3807b644-377c-4da5-a3ff-07d16c0972d2`::Granted request >Thread-1909::DEBUG::2012-08-30 13:03:41,104::task::817::TaskManager.Task::(resourceAcquired) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1909::DEBUG::2012-08-30 13:03:41,104::task::978::TaskManager.Task::(_decref) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::ref 1 aborting False >Thread-1909::INFO::2012-08-30 13:03:41,107::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1909::DEBUG::2012-08-30 13:03:41,107::task::1172::TaskManager.Task::(prepare) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1909::DEBUG::2012-08-30 13:03:41,107::task::588::TaskManager.Task::(_updateState) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::moving from state preparing -> state finished >Thread-1909::DEBUG::2012-08-30 13:03:41,108::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1909::DEBUG::2012-08-30 13:03:41,108::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1909::DEBUG::2012-08-30 13:03:41,108::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1909::DEBUG::2012-08-30 13:03:41,108::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1909::DEBUG::2012-08-30 13:03:41,109::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1909::DEBUG::2012-08-30 13:03:41,109::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1909::DEBUG::2012-08-30 13:03:41,109::task::978::TaskManager.Task::(_decref) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:03:43,395::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:45,426::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:45,426::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:48,396::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:03:50,427::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:03:53,397::task::588::TaskManager.Task::(_updateState) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:03:53,398::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:03:53,398::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`5a856057-1a02-4ecf-919d-340c96127d1a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:03:53,398::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:03:53,398::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:03:53,399::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`5a856057-1a02-4ecf-919d-340c96127d1a`::Granted request >Thread-40::DEBUG::2012-08-30 13:03:53,399::task::817::TaskManager.Task::(resourceAcquired) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:03:53,399::task::978::TaskManager.Task::(_decref) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:03:53,401::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:03:53,403::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:03:53,404::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:03:53,404::task::1172::TaskManager.Task::(prepare) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:03:53,405::task::588::TaskManager.Task::(_updateState) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:03:53,405::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:03:53,405::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:03:53,405::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:03:53,406::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:03:53,406::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:03:53,406::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:03:53,406::task::978::TaskManager.Task::(_decref) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:03:53,407::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:53,407::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:53,408::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:03:53,408::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1912::DEBUG::2012-08-30 13:03:54,439::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1913::DEBUG::2012-08-30 13:03:54,440::task::588::TaskManager.Task::(_updateState) Task=`9b71abda-beba-4be3-b74b-dd17e2f83288`::moving from state init -> state preparing >Thread-1912::DEBUG::2012-08-30 13:03:54,440::task::588::TaskManager.Task::(_updateState) Task=`98a2c98e-f450-4c1f-a738-70e14513d2a6`::moving from state init -> state preparing >Thread-1913::INFO::2012-08-30 13:03:54,440::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1912::INFO::2012-08-30 13:03:54,440::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1913::INFO::2012-08-30 13:03:54,441::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176906585693', 'lastCheck': 1346303030.3632259, 'code': 0, 'valid': True}} >Thread-1912::INFO::2012-08-30 13:03:54,441::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1913::DEBUG::2012-08-30 13:03:54,441::task::1172::TaskManager.Task::(prepare) Task=`9b71abda-beba-4be3-b74b-dd17e2f83288`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176906585693', 'lastCheck': 1346303030.3632259, 'code': 0, 'valid': True}} >Thread-1912::DEBUG::2012-08-30 13:03:54,441::task::1172::TaskManager.Task::(prepare) Task=`98a2c98e-f450-4c1f-a738-70e14513d2a6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1913::DEBUG::2012-08-30 13:03:54,442::task::588::TaskManager.Task::(_updateState) Task=`9b71abda-beba-4be3-b74b-dd17e2f83288`::moving from state preparing -> state finished >Thread-1912::DEBUG::2012-08-30 13:03:54,442::task::588::TaskManager.Task::(_updateState) Task=`98a2c98e-f450-4c1f-a738-70e14513d2a6`::moving from state preparing -> state finished >Thread-1913::DEBUG::2012-08-30 13:03:54,442::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1912::DEBUG::2012-08-30 13:03:54,442::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1913::DEBUG::2012-08-30 13:03:54,442::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1912::DEBUG::2012-08-30 13:03:54,442::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1913::DEBUG::2012-08-30 13:03:54,443::task::978::TaskManager.Task::(_decref) Task=`9b71abda-beba-4be3-b74b-dd17e2f83288`::ref 0 aborting False >Thread-1912::DEBUG::2012-08-30 13:03:54,443::task::978::TaskManager.Task::(_decref) Task=`98a2c98e-f450-4c1f-a738-70e14513d2a6`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:03:55,428::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:03:55,723::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1914::DEBUG::2012-08-30 13:03:57,596::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1914::DEBUG::2012-08-30 13:03:57,596::task::588::TaskManager.Task::(_updateState) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::moving from state init -> state preparing >Thread-1914::INFO::2012-08-30 13:03:57,596::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1914::DEBUG::2012-08-30 13:03:57,597::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0cc7179-ade4-4f5d-9024-4c0459482ca5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1914::DEBUG::2012-08-30 13:03:57,597::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1914::DEBUG::2012-08-30 13:03:57,597::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1914::DEBUG::2012-08-30 13:03:57,597::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0cc7179-ade4-4f5d-9024-4c0459482ca5`::Granted request >Thread-1914::DEBUG::2012-08-30 13:03:57,598::task::817::TaskManager.Task::(resourceAcquired) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1914::DEBUG::2012-08-30 13:03:57,598::task::978::TaskManager.Task::(_decref) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::ref 1 aborting False >Thread-1915::DEBUG::2012-08-30 13:03:57,599::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1915::DEBUG::2012-08-30 13:03:57,599::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1914::INFO::2012-08-30 13:03:57,603::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1914::DEBUG::2012-08-30 13:03:57,603::task::1172::TaskManager.Task::(prepare) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1914::DEBUG::2012-08-30 13:03:57,604::task::588::TaskManager.Task::(_updateState) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::moving from state preparing -> state finished >Thread-1914::DEBUG::2012-08-30 13:03:57,604::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1914::DEBUG::2012-08-30 13:03:57,604::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1914::DEBUG::2012-08-30 13:03:57,605::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1914::DEBUG::2012-08-30 13:03:57,605::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1914::DEBUG::2012-08-30 13:03:57,605::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1914::DEBUG::2012-08-30 13:03:57,605::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1914::DEBUG::2012-08-30 13:03:57,605::task::978::TaskManager.Task::(_decref) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:03:58,409::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:00,429::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:00,429::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:04:02,730::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:04:03,410::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:05,430::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:08,412::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:08,412::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:10,431::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1918::DEBUG::2012-08-30 13:04:10,816::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1918::DEBUG::2012-08-30 13:04:10,816::task::588::TaskManager.Task::(_updateState) Task=`4ce3b2c5-d4fc-4ed6-9eaf-546e86bd2ee2`::moving from state init -> state preparing >Thread-1918::INFO::2012-08-30 13:04:10,817::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1918::INFO::2012-08-30 13:04:10,817::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1918::DEBUG::2012-08-30 13:04:10,817::task::1172::TaskManager.Task::(prepare) Task=`4ce3b2c5-d4fc-4ed6-9eaf-546e86bd2ee2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1918::DEBUG::2012-08-30 13:04:10,817::task::588::TaskManager.Task::(_updateState) Task=`4ce3b2c5-d4fc-4ed6-9eaf-546e86bd2ee2`::moving from state preparing -> state finished >Thread-1918::DEBUG::2012-08-30 13:04:10,818::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1918::DEBUG::2012-08-30 13:04:10,818::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1918::DEBUG::2012-08-30 13:04:10,818::task::978::TaskManager.Task::(_decref) Task=`4ce3b2c5-d4fc-4ed6-9eaf-546e86bd2ee2`::ref 0 aborting False >Thread-1920::DEBUG::2012-08-30 13:04:13,349::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1920::DEBUG::2012-08-30 13:04:13,350::task::588::TaskManager.Task::(_updateState) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::moving from state init -> state preparing >Thread-1920::INFO::2012-08-30 13:04:13,351::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1920::DEBUG::2012-08-30 13:04:13,351::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5f4972ec-1136-4872-b821-01441689ec19`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1920::DEBUG::2012-08-30 13:04:13,351::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1920::DEBUG::2012-08-30 13:04:13,351::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1920::DEBUG::2012-08-30 13:04:13,352::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5f4972ec-1136-4872-b821-01441689ec19`::Granted request >Thread-1920::DEBUG::2012-08-30 13:04:13,352::task::817::TaskManager.Task::(resourceAcquired) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1920::DEBUG::2012-08-30 13:04:13,352::task::978::TaskManager.Task::(_decref) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::ref 1 aborting False >Thread-1920::INFO::2012-08-30 13:04:13,355::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1920::DEBUG::2012-08-30 13:04:13,355::task::1172::TaskManager.Task::(prepare) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1920::DEBUG::2012-08-30 13:04:13,356::task::588::TaskManager.Task::(_updateState) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::moving from state preparing -> state finished >Thread-1920::DEBUG::2012-08-30 13:04:13,356::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1920::DEBUG::2012-08-30 13:04:13,356::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1920::DEBUG::2012-08-30 13:04:13,356::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1920::DEBUG::2012-08-30 13:04:13,357::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1920::DEBUG::2012-08-30 13:04:13,357::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1920::DEBUG::2012-08-30 13:04:13,357::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1920::DEBUG::2012-08-30 13:04:13,357::task::978::TaskManager.Task::(_decref) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:04:13,413::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:15,432::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:15,433::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:18,414::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:20,433::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:23,416::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:23,416::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1922::DEBUG::2012-08-30 13:04:23,821::task::588::TaskManager.Task::(_updateState) Task=`077ded06-397b-41e8-844d-44fc518841d1`::moving from state init -> state preparing >Thread-1922::INFO::2012-08-30 13:04:23,822::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1922::INFO::2012-08-30 13:04:23,822::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162315368652', 'lastCheck': 1346303060.383126, 'code': 0, 'valid': True}} >Thread-1922::DEBUG::2012-08-30 13:04:23,822::task::1172::TaskManager.Task::(prepare) Task=`077ded06-397b-41e8-844d-44fc518841d1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162315368652', 'lastCheck': 1346303060.383126, 'code': 0, 'valid': True}} >Thread-1922::DEBUG::2012-08-30 13:04:23,822::task::588::TaskManager.Task::(_updateState) Task=`077ded06-397b-41e8-844d-44fc518841d1`::moving from state preparing -> state finished >Thread-1922::DEBUG::2012-08-30 13:04:23,823::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1922::DEBUG::2012-08-30 13:04:23,823::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1922::DEBUG::2012-08-30 13:04:23,823::task::978::TaskManager.Task::(_decref) Task=`077ded06-397b-41e8-844d-44fc518841d1`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:04:25,434::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:04:25,750::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1924::DEBUG::2012-08-30 13:04:26,627::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1923::DEBUG::2012-08-30 13:04:26,627::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1924::DEBUG::2012-08-30 13:04:26,627::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-1923::DEBUG::2012-08-30 13:04:26,628::task::588::TaskManager.Task::(_updateState) Task=`9cc5f57e-d8a7-488f-90c6-d50e3eb0a097`::moving from state init -> state preparing >Thread-1923::INFO::2012-08-30 13:04:26,629::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1923::INFO::2012-08-30 13:04:26,629::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1923::DEBUG::2012-08-30 13:04:26,630::task::1172::TaskManager.Task::(prepare) Task=`9cc5f57e-d8a7-488f-90c6-d50e3eb0a097`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1923::DEBUG::2012-08-30 13:04:26,630::task::588::TaskManager.Task::(_updateState) Task=`9cc5f57e-d8a7-488f-90c6-d50e3eb0a097`::moving from state preparing -> state finished >Thread-1923::DEBUG::2012-08-30 13:04:26,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1923::DEBUG::2012-08-30 13:04:26,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1923::DEBUG::2012-08-30 13:04:26,631::task::978::TaskManager.Task::(_decref) Task=`9cc5f57e-d8a7-488f-90c6-d50e3eb0a097`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:04:28,417::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1925::DEBUG::2012-08-30 13:04:29,783::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1925::DEBUG::2012-08-30 13:04:29,784::task::588::TaskManager.Task::(_updateState) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::moving from state init -> state preparing >Thread-1925::INFO::2012-08-30 13:04:29,784::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1925::DEBUG::2012-08-30 13:04:29,784::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`af8e09dc-d20f-4f9f-8d2f-9785c56aa197`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1925::DEBUG::2012-08-30 13:04:29,785::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1925::DEBUG::2012-08-30 13:04:29,785::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1925::DEBUG::2012-08-30 13:04:29,785::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`af8e09dc-d20f-4f9f-8d2f-9785c56aa197`::Granted request >Thread-1925::DEBUG::2012-08-30 13:04:29,785::task::817::TaskManager.Task::(resourceAcquired) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1925::DEBUG::2012-08-30 13:04:29,786::task::978::TaskManager.Task::(_decref) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::ref 1 aborting False >Thread-1925::INFO::2012-08-30 13:04:29,789::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1925::DEBUG::2012-08-30 13:04:29,789::task::1172::TaskManager.Task::(prepare) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1925::DEBUG::2012-08-30 13:04:29,789::task::588::TaskManager.Task::(_updateState) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::moving from state preparing -> state finished >Thread-1925::DEBUG::2012-08-30 13:04:29,789::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1925::DEBUG::2012-08-30 13:04:29,791::task::978::TaskManager.Task::(_decref) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 13:04:30,435::task::588::TaskManager.Task::(_updateState) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:04:30,436::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:04:30,436::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`15e13fd9-75d4-439c-889a-7de5d22808a4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:04:30,436::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:04:30,437::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:04:30,437::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`15e13fd9-75d4-439c-889a-7de5d22808a4`::Granted request >Thread-104::DEBUG::2012-08-30 13:04:30,437::task::817::TaskManager.Task::(resourceAcquired) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:04:30,437::task::978::TaskManager.Task::(_decref) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:04:30,439::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:04:30,441::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:04:30,442::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:04:30,442::task::1172::TaskManager.Task::(prepare) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:04:30,442::task::588::TaskManager.Task::(_updateState) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:04:30,443::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:04:30,443::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:04:30,443::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:04:30,443::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:04:30,444::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:04:30,444::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:04:30,444::task::978::TaskManager.Task::(_decref) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:04:30,445::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:30,445::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:30,445::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:30,446::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:04:32,757::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:04:33,418::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:35,447::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:38,419::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:38,420::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:40,448::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1929::DEBUG::2012-08-30 13:04:42,985::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1929::DEBUG::2012-08-30 13:04:42,986::task::588::TaskManager.Task::(_updateState) Task=`b5c3068e-220a-40cf-be44-6f1b4da94127`::moving from state init -> state preparing >Thread-1929::INFO::2012-08-30 13:04:42,986::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1929::INFO::2012-08-30 13:04:42,986::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1929::DEBUG::2012-08-30 13:04:42,986::task::1172::TaskManager.Task::(prepare) Task=`b5c3068e-220a-40cf-be44-6f1b4da94127`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1929::DEBUG::2012-08-30 13:04:42,987::task::588::TaskManager.Task::(_updateState) Task=`b5c3068e-220a-40cf-be44-6f1b4da94127`::moving from state preparing -> state finished >Thread-1929::DEBUG::2012-08-30 13:04:42,987::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1929::DEBUG::2012-08-30 13:04:42,987::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1929::DEBUG::2012-08-30 13:04:42,987::task::978::TaskManager.Task::(_decref) Task=`b5c3068e-220a-40cf-be44-6f1b4da94127`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:04:43,421::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:45,449::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:45,450::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1930::DEBUG::2012-08-30 13:04:46,125::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1930::DEBUG::2012-08-30 13:04:46,126::task::588::TaskManager.Task::(_updateState) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::moving from state init -> state preparing >Thread-1930::INFO::2012-08-30 13:04:46,126::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1930::DEBUG::2012-08-30 13:04:46,126::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0cdb43c9-e7a5-4761-ba31-9f46e0767443`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1930::DEBUG::2012-08-30 13:04:46,126::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1930::DEBUG::2012-08-30 13:04:46,127::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1930::DEBUG::2012-08-30 13:04:46,127::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0cdb43c9-e7a5-4761-ba31-9f46e0767443`::Granted request >Thread-1930::DEBUG::2012-08-30 13:04:46,127::task::817::TaskManager.Task::(resourceAcquired) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1930::DEBUG::2012-08-30 13:04:46,128::task::978::TaskManager.Task::(_decref) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::ref 1 aborting False >Thread-1930::INFO::2012-08-30 13:04:46,130::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1930::DEBUG::2012-08-30 13:04:46,131::task::1172::TaskManager.Task::(prepare) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1930::DEBUG::2012-08-30 13:04:46,131::task::588::TaskManager.Task::(_updateState) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::moving from state preparing -> state finished >Thread-1930::DEBUG::2012-08-30 13:04:46,131::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1930::DEBUG::2012-08-30 13:04:46,131::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1930::DEBUG::2012-08-30 13:04:46,132::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1930::DEBUG::2012-08-30 13:04:46,132::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1930::DEBUG::2012-08-30 13:04:46,132::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1930::DEBUG::2012-08-30 13:04:46,132::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1930::DEBUG::2012-08-30 13:04:46,133::task::978::TaskManager.Task::(_decref) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:04:48,421::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:04:50,451::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1932::DEBUG::2012-08-30 13:04:51,273::task::588::TaskManager.Task::(_updateState) Task=`ed585a9a-7fec-4320-a43c-1d1345a802d8`::moving from state init -> state preparing >Thread-1932::INFO::2012-08-30 13:04:51,273::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1932::INFO::2012-08-30 13:04:51,273::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177884101868', 'lastCheck': 1346303090.4032381, 'code': 0, 'valid': True}} >Thread-1932::DEBUG::2012-08-30 13:04:51,274::task::1172::TaskManager.Task::(prepare) Task=`ed585a9a-7fec-4320-a43c-1d1345a802d8`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177884101868', 'lastCheck': 1346303090.4032381, 'code': 0, 'valid': True}} >Thread-1932::DEBUG::2012-08-30 13:04:51,274::task::588::TaskManager.Task::(_updateState) Task=`ed585a9a-7fec-4320-a43c-1d1345a802d8`::moving from state preparing -> state finished >Thread-1932::DEBUG::2012-08-30 13:04:51,274::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1932::DEBUG::2012-08-30 13:04:51,274::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1932::DEBUG::2012-08-30 13:04:51,274::task::978::TaskManager.Task::(_decref) Task=`ed585a9a-7fec-4320-a43c-1d1345a802d8`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 13:04:53,422::task::588::TaskManager.Task::(_updateState) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:04:53,423::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:04:53,423::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ccf26099-948f-4506-ae65-3c708e974e34`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:04:53,423::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:04:53,424::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:04:53,424::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ccf26099-948f-4506-ae65-3c708e974e34`::Granted request >Thread-40::DEBUG::2012-08-30 13:04:53,424::task::817::TaskManager.Task::(resourceAcquired) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:04:53,425::task::978::TaskManager.Task::(_decref) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:04:53,426::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:04:53,428::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:04:53,430::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:04:53,430::task::1172::TaskManager.Task::(prepare) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:04:53,430::task::588::TaskManager.Task::(_updateState) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:04:53,430::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:04:53,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:04:53,431::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:04:53,431::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:04:53,431::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:04:53,431::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:04:53,432::task::978::TaskManager.Task::(_decref) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:04:53,432::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:53,433::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:53,433::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:04:53,433::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1933::DEBUG::2012-08-30 13:04:54,415::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1933::DEBUG::2012-08-30 13:04:54,415::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:04:55,452::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:04:55,775::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:04:58,434::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1934::DEBUG::2012-08-30 13:04:59,352::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1934::DEBUG::2012-08-30 13:04:59,352::task::588::TaskManager.Task::(_updateState) Task=`21395c7c-c90e-4965-8416-26950534b187`::moving from state init -> state preparing >Thread-1934::INFO::2012-08-30 13:04:59,352::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1934::INFO::2012-08-30 13:04:59,353::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1934::DEBUG::2012-08-30 13:04:59,353::task::1172::TaskManager.Task::(prepare) Task=`21395c7c-c90e-4965-8416-26950534b187`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1934::DEBUG::2012-08-30 13:04:59,353::task::588::TaskManager.Task::(_updateState) Task=`21395c7c-c90e-4965-8416-26950534b187`::moving from state preparing -> state finished >Thread-1934::DEBUG::2012-08-30 13:04:59,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1934::DEBUG::2012-08-30 13:04:59,354::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1934::DEBUG::2012-08-30 13:04:59,354::task::978::TaskManager.Task::(_decref) Task=`21395c7c-c90e-4965-8416-26950534b187`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:05:00,454::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:00,454::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1936::DEBUG::2012-08-30 13:05:02,492::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1936::DEBUG::2012-08-30 13:05:02,492::task::588::TaskManager.Task::(_updateState) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::moving from state init -> state preparing >Thread-1936::INFO::2012-08-30 13:05:02,493::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1936::DEBUG::2012-08-30 13:05:02,493::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2c81b8e6-2905-4d49-9bf8-abca1bad3fe9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1936::DEBUG::2012-08-30 13:05:02,493::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1936::DEBUG::2012-08-30 13:05:02,494::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1936::DEBUG::2012-08-30 13:05:02,494::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2c81b8e6-2905-4d49-9bf8-abca1bad3fe9`::Granted request >Thread-1936::DEBUG::2012-08-30 13:05:02,495::task::817::TaskManager.Task::(resourceAcquired) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1936::DEBUG::2012-08-30 13:05:02,495::task::978::TaskManager.Task::(_decref) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::ref 1 aborting False >Thread-1936::INFO::2012-08-30 13:05:02,498::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1936::DEBUG::2012-08-30 13:05:02,498::task::1172::TaskManager.Task::(prepare) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1936::DEBUG::2012-08-30 13:05:02,498::task::588::TaskManager.Task::(_updateState) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::moving from state preparing -> state finished >Thread-1936::DEBUG::2012-08-30 13:05:02,498::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1936::DEBUG::2012-08-30 13:05:02,499::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1936::DEBUG::2012-08-30 13:05:02,499::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1936::DEBUG::2012-08-30 13:05:02,499::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1936::DEBUG::2012-08-30 13:05:02,499::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1936::DEBUG::2012-08-30 13:05:02,500::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1936::DEBUG::2012-08-30 13:05:02,500::task::978::TaskManager.Task::(_decref) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:05:02,779::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:05:03,435::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:05,455::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:08,436::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:08,437::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:10,456::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:13,438::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:15,458::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:15,458::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1940::DEBUG::2012-08-30 13:05:15,680::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1940::DEBUG::2012-08-30 13:05:15,680::task::588::TaskManager.Task::(_updateState) Task=`50b5a93f-eeb3-4bd4-8463-2e6fb0822766`::moving from state init -> state preparing >Thread-1940::INFO::2012-08-30 13:05:15,680::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1940::INFO::2012-08-30 13:05:15,681::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1940::DEBUG::2012-08-30 13:05:15,681::task::1172::TaskManager.Task::(prepare) Task=`50b5a93f-eeb3-4bd4-8463-2e6fb0822766`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1940::DEBUG::2012-08-30 13:05:15,681::task::588::TaskManager.Task::(_updateState) Task=`50b5a93f-eeb3-4bd4-8463-2e6fb0822766`::moving from state preparing -> state finished >Thread-1940::DEBUG::2012-08-30 13:05:15,681::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1940::DEBUG::2012-08-30 13:05:15,682::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1940::DEBUG::2012-08-30 13:05:15,682::task::978::TaskManager.Task::(_decref) Task=`50b5a93f-eeb3-4bd4-8463-2e6fb0822766`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:05:18,439::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1941::DEBUG::2012-08-30 13:05:18,820::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1942::DEBUG::2012-08-30 13:05:18,821::task::588::TaskManager.Task::(_updateState) Task=`f5d36a44-3d9d-4ca4-8330-e04024f6fec4`::moving from state init -> state preparing >Thread-1941::DEBUG::2012-08-30 13:05:18,821::task::588::TaskManager.Task::(_updateState) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::moving from state init -> state preparing >Thread-1942::INFO::2012-08-30 13:05:18,821::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1941::INFO::2012-08-30 13:05:18,821::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1942::INFO::2012-08-30 13:05:18,822::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00266218185425', 'lastCheck': 1346303110.418643, 'code': 0, 'valid': True}} >Thread-1941::DEBUG::2012-08-30 13:05:18,822::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`76cf2168-d497-4f1d-92b6-7f3fc519d000`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1942::DEBUG::2012-08-30 13:05:18,822::task::1172::TaskManager.Task::(prepare) Task=`f5d36a44-3d9d-4ca4-8330-e04024f6fec4`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00266218185425', 'lastCheck': 1346303110.418643, 'code': 0, 'valid': True}} >Thread-1941::DEBUG::2012-08-30 13:05:18,822::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1942::DEBUG::2012-08-30 13:05:18,823::task::588::TaskManager.Task::(_updateState) Task=`f5d36a44-3d9d-4ca4-8330-e04024f6fec4`::moving from state preparing -> state finished >Thread-1941::DEBUG::2012-08-30 13:05:18,823::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1942::DEBUG::2012-08-30 13:05:18,823::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1941::DEBUG::2012-08-30 13:05:18,823::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`76cf2168-d497-4f1d-92b6-7f3fc519d000`::Granted request >Thread-1942::DEBUG::2012-08-30 13:05:18,823::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1941::DEBUG::2012-08-30 13:05:18,824::task::817::TaskManager.Task::(resourceAcquired) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1942::DEBUG::2012-08-30 13:05:18,824::task::978::TaskManager.Task::(_decref) Task=`f5d36a44-3d9d-4ca4-8330-e04024f6fec4`::ref 0 aborting False >Thread-1941::DEBUG::2012-08-30 13:05:18,824::task::978::TaskManager.Task::(_decref) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::ref 1 aborting False >Thread-1941::INFO::2012-08-30 13:05:18,829::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1941::DEBUG::2012-08-30 13:05:18,829::task::1172::TaskManager.Task::(prepare) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1941::DEBUG::2012-08-30 13:05:18,829::task::588::TaskManager.Task::(_updateState) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::moving from state preparing -> state finished >Thread-1941::DEBUG::2012-08-30 13:05:18,830::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1941::DEBUG::2012-08-30 13:05:18,830::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1941::DEBUG::2012-08-30 13:05:18,830::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1941::DEBUG::2012-08-30 13:05:18,830::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1941::DEBUG::2012-08-30 13:05:18,831::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1941::DEBUG::2012-08-30 13:05:18,831::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1941::DEBUG::2012-08-30 13:05:18,831::task::978::TaskManager.Task::(_decref) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:05:20,459::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1943::DEBUG::2012-08-30 13:05:21,966::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1943::DEBUG::2012-08-30 13:05:21,967::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:05:23,440::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:23,440::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:25,461::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:05:25,798::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:05:28,441::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-26::DEBUG::2012-08-30 13:05:30,426::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:05:30,426::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-26::DEBUG::2012-08-30 13:05:30,427::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None) >Thread-104::DEBUG::2012-08-30 13:05:30,462::task::588::TaskManager.Task::(_updateState) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:05:30,463::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:05:30,463::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`216b723a-03a8-424c-bec4-1091afa4c5d5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:05:30,463::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:05:30,464::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:05:30,464::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`216b723a-03a8-424c-bec4-1091afa4c5d5`::Granted request >Thread-104::DEBUG::2012-08-30 13:05:30,464::task::817::TaskManager.Task::(resourceAcquired) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:05:30,465::task::978::TaskManager.Task::(_decref) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::ref 1 aborting False >Thread-26::DEBUG::2012-08-30 13:05:30,541::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5 >Thread-26::WARNING::2012-08-30 13:05:30,541::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found'] >Thread-26::DEBUG::2012-08-30 13:05:30,542::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-26::DEBUG::2012-08-30 13:05:30,548::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:05:30,548::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend >Thread-26::DEBUG::2012-08-30 13:05:30,550::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d'] >Thread-26::DEBUG::2012-08-30 13:05:30,552::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-26::WARNING::2012-08-30 13:05:30,552::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered >Thread-26::WARNING::2012-08-30 13:05:30,552::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered >Thread-104::DEBUG::2012-08-30 13:05:30,554::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:05:30,564::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:05:30,566::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:05:30,566::task::1172::TaskManager.Task::(prepare) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:05:30,566::task::588::TaskManager.Task::(_updateState) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:05:30,567::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:05:30,567::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:05:30,567::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:05:30,568::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:05:30,568::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:05:30,568::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:05:30,568::task::978::TaskManager.Task::(_decref) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:05:30,569::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:30,569::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:30,570::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:30,570::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1945::DEBUG::2012-08-30 13:05:32,023::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1945::DEBUG::2012-08-30 13:05:32,024::task::588::TaskManager.Task::(_updateState) Task=`67dd5280-d24f-491c-b08f-ac2ad2853152`::moving from state init -> state preparing >Thread-1945::INFO::2012-08-30 13:05:32,024::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1945::INFO::2012-08-30 13:05:32,024::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1945::DEBUG::2012-08-30 13:05:32,025::task::1172::TaskManager.Task::(prepare) Task=`67dd5280-d24f-491c-b08f-ac2ad2853152`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1945::DEBUG::2012-08-30 13:05:32,025::task::588::TaskManager.Task::(_updateState) Task=`67dd5280-d24f-491c-b08f-ac2ad2853152`::moving from state preparing -> state finished >Thread-1945::DEBUG::2012-08-30 13:05:32,025::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1945::DEBUG::2012-08-30 13:05:32,025::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1945::DEBUG::2012-08-30 13:05:32,025::task::978::TaskManager.Task::(_decref) Task=`67dd5280-d24f-491c-b08f-ac2ad2853152`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:05:32,805::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:05:33,442::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1952::DEBUG::2012-08-30 13:05:35,312::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1952::DEBUG::2012-08-30 13:05:35,312::task::588::TaskManager.Task::(_updateState) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::moving from state init -> state preparing >Thread-1952::INFO::2012-08-30 13:05:35,312::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1952::DEBUG::2012-08-30 13:05:35,313::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f6ad3412-0190-4815-9886-f8e13d60d9b2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1952::DEBUG::2012-08-30 13:05:35,313::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1952::DEBUG::2012-08-30 13:05:35,313::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1952::DEBUG::2012-08-30 13:05:35,313::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f6ad3412-0190-4815-9886-f8e13d60d9b2`::Granted request >Thread-1952::DEBUG::2012-08-30 13:05:35,314::task::817::TaskManager.Task::(resourceAcquired) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1952::DEBUG::2012-08-30 13:05:35,314::task::978::TaskManager.Task::(_decref) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::ref 1 aborting False >Thread-1952::INFO::2012-08-30 13:05:35,317::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1952::DEBUG::2012-08-30 13:05:35,317::task::1172::TaskManager.Task::(prepare) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1952::DEBUG::2012-08-30 13:05:35,318::task::588::TaskManager.Task::(_updateState) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::moving from state preparing -> state finished >Thread-1952::DEBUG::2012-08-30 13:05:35,318::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1952::DEBUG::2012-08-30 13:05:35,318::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1952::DEBUG::2012-08-30 13:05:35,318::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1952::DEBUG::2012-08-30 13:05:35,319::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1952::DEBUG::2012-08-30 13:05:35,319::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1952::DEBUG::2012-08-30 13:05:35,319::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1952::DEBUG::2012-08-30 13:05:35,319::task::978::TaskManager.Task::(_decref) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:05:35,572::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:38,443::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:38,444::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:40,573::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:43,444::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:45,574::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:45,575::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1955::DEBUG::2012-08-30 13:05:46,989::task::588::TaskManager.Task::(_updateState) Task=`1b46c342-9527-41b6-8fb5-6140cdd45304`::moving from state init -> state preparing >Thread-1955::INFO::2012-08-30 13:05:46,990::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1955::INFO::2012-08-30 13:05:46,990::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303140.610781, 'code': 0, 'valid': True}} >Thread-1955::DEBUG::2012-08-30 13:05:46,990::task::1172::TaskManager.Task::(prepare) Task=`1b46c342-9527-41b6-8fb5-6140cdd45304`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303140.610781, 'code': 0, 'valid': True}} >Thread-1955::DEBUG::2012-08-30 13:05:46,991::task::588::TaskManager.Task::(_updateState) Task=`1b46c342-9527-41b6-8fb5-6140cdd45304`::moving from state preparing -> state finished >Thread-1955::DEBUG::2012-08-30 13:05:46,991::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1955::DEBUG::2012-08-30 13:05:46,991::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1955::DEBUG::2012-08-30 13:05:46,991::task::978::TaskManager.Task::(_decref) Task=`1b46c342-9527-41b6-8fb5-6140cdd45304`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:05:48,445::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1956::DEBUG::2012-08-30 13:05:48,564::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1956::DEBUG::2012-08-30 13:05:48,565::task::588::TaskManager.Task::(_updateState) Task=`3394e19f-2c77-461c-bd06-bbcdf348c612`::moving from state init -> state preparing >Thread-1956::INFO::2012-08-30 13:05:48,565::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1956::INFO::2012-08-30 13:05:48,565::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1956::DEBUG::2012-08-30 13:05:48,566::task::1172::TaskManager.Task::(prepare) Task=`3394e19f-2c77-461c-bd06-bbcdf348c612`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1956::DEBUG::2012-08-30 13:05:48,566::task::588::TaskManager.Task::(_updateState) Task=`3394e19f-2c77-461c-bd06-bbcdf348c612`::moving from state preparing -> state finished >Thread-1956::DEBUG::2012-08-30 13:05:48,566::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1956::DEBUG::2012-08-30 13:05:48,566::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1956::DEBUG::2012-08-30 13:05:48,566::task::978::TaskManager.Task::(_decref) Task=`3394e19f-2c77-461c-bd06-bbcdf348c612`::ref 0 aborting False >Thread-1957::DEBUG::2012-08-30 13:05:50,139::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1957::DEBUG::2012-08-30 13:05:50,140::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:05:50,576::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1958::DEBUG::2012-08-30 13:05:51,712::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1958::DEBUG::2012-08-30 13:05:51,713::task::588::TaskManager.Task::(_updateState) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::moving from state init -> state preparing >Thread-1958::INFO::2012-08-30 13:05:51,713::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1958::DEBUG::2012-08-30 13:05:51,713::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e44b5af8-850a-40f8-8bae-ca6c94d3e8f2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1958::DEBUG::2012-08-30 13:05:51,714::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1958::DEBUG::2012-08-30 13:05:51,714::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1958::DEBUG::2012-08-30 13:05:51,714::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e44b5af8-850a-40f8-8bae-ca6c94d3e8f2`::Granted request >Thread-1958::DEBUG::2012-08-30 13:05:51,714::task::817::TaskManager.Task::(resourceAcquired) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1958::DEBUG::2012-08-30 13:05:51,715::task::978::TaskManager.Task::(_decref) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::ref 1 aborting False >Thread-1958::INFO::2012-08-30 13:05:51,718::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1958::DEBUG::2012-08-30 13:05:51,718::task::1172::TaskManager.Task::(prepare) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1958::DEBUG::2012-08-30 13:05:51,718::task::588::TaskManager.Task::(_updateState) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::moving from state preparing -> state finished >Thread-1958::DEBUG::2012-08-30 13:05:51,718::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1958::DEBUG::2012-08-30 13:05:51,718::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1958::DEBUG::2012-08-30 13:05:51,719::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1958::DEBUG::2012-08-30 13:05:51,719::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1958::DEBUG::2012-08-30 13:05:51,719::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1958::DEBUG::2012-08-30 13:05:51,719::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1958::DEBUG::2012-08-30 13:05:51,720::task::978::TaskManager.Task::(_decref) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 13:05:53,446::task::588::TaskManager.Task::(_updateState) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:05:53,447::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:05:53,447::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f1892dc5-144e-4c10-bdbf-cbacd35db739`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:05:53,447::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:05:53,448::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:05:53,448::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f1892dc5-144e-4c10-bdbf-cbacd35db739`::Granted request >Thread-40::DEBUG::2012-08-30 13:05:53,448::task::817::TaskManager.Task::(resourceAcquired) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:05:53,448::task::978::TaskManager.Task::(_decref) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:05:53,450::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:05:53,452::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:05:53,453::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:05:53,454::task::1172::TaskManager.Task::(prepare) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:05:53,454::task::588::TaskManager.Task::(_updateState) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:05:53,454::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:05:53,454::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:05:53,455::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:05:53,455::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:05:53,455::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:05:53,455::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:05:53,456::task::978::TaskManager.Task::(_decref) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:05:53,456::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:53,456::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:53,457::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:05:53,457::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:05:55,577::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:05:55,824::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:05:58,458::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:00,578::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:00,579::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:06:02,829::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:06:03,459::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:05,580::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1961::DEBUG::2012-08-30 13:06:06,373::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1961::DEBUG::2012-08-30 13:06:06,374::task::588::TaskManager.Task::(_updateState) Task=`866af8c6-c747-4add-9634-ceb69fc447bf`::moving from state init -> state preparing >Thread-1961::INFO::2012-08-30 13:06:06,374::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1961::INFO::2012-08-30 13:06:06,374::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1961::DEBUG::2012-08-30 13:06:06,374::task::1172::TaskManager.Task::(prepare) Task=`866af8c6-c747-4add-9634-ceb69fc447bf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1961::DEBUG::2012-08-30 13:06:06,375::task::588::TaskManager.Task::(_updateState) Task=`866af8c6-c747-4add-9634-ceb69fc447bf`::moving from state preparing -> state finished >Thread-1961::DEBUG::2012-08-30 13:06:06,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1961::DEBUG::2012-08-30 13:06:06,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1961::DEBUG::2012-08-30 13:06:06,375::task::978::TaskManager.Task::(_decref) Task=`866af8c6-c747-4add-9634-ceb69fc447bf`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:06:08,460::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:08,460::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1963::DEBUG::2012-08-30 13:06:09,562::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1963::DEBUG::2012-08-30 13:06:09,562::task::588::TaskManager.Task::(_updateState) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::moving from state init -> state preparing >Thread-1963::INFO::2012-08-30 13:06:09,563::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1963::DEBUG::2012-08-30 13:06:09,563::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8977a339-343f-47af-ba27-544ee7c79a4e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1963::DEBUG::2012-08-30 13:06:09,563::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1963::DEBUG::2012-08-30 13:06:09,563::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1963::DEBUG::2012-08-30 13:06:09,564::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8977a339-343f-47af-ba27-544ee7c79a4e`::Granted request >Thread-1963::DEBUG::2012-08-30 13:06:09,564::task::817::TaskManager.Task::(resourceAcquired) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1963::DEBUG::2012-08-30 13:06:09,564::task::978::TaskManager.Task::(_decref) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::ref 1 aborting False >Thread-1963::INFO::2012-08-30 13:06:09,567::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1963::DEBUG::2012-08-30 13:06:09,567::task::1172::TaskManager.Task::(prepare) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1963::DEBUG::2012-08-30 13:06:09,568::task::588::TaskManager.Task::(_updateState) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::moving from state preparing -> state finished >Thread-1963::DEBUG::2012-08-30 13:06:09,568::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1963::DEBUG::2012-08-30 13:06:09,568::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1963::DEBUG::2012-08-30 13:06:09,568::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1963::DEBUG::2012-08-30 13:06:09,569::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1963::DEBUG::2012-08-30 13:06:09,569::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1963::DEBUG::2012-08-30 13:06:09,569::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1963::DEBUG::2012-08-30 13:06:09,569::task::978::TaskManager.Task::(_decref) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:06:10,581::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:13,461::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:15,582::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:15,583::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1965::DEBUG::2012-08-30 13:06:17,545::task::588::TaskManager.Task::(_updateState) Task=`b53d6443-efe4-4af1-9d45-50a7c90f6000`::moving from state init -> state preparing >Thread-1965::INFO::2012-08-30 13:06:17,545::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1965::INFO::2012-08-30 13:06:17,546::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303170.631319, 'code': 0, 'valid': True}} >Thread-1965::DEBUG::2012-08-30 13:06:17,546::task::1172::TaskManager.Task::(prepare) Task=`b53d6443-efe4-4af1-9d45-50a7c90f6000`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303170.631319, 'code': 0, 'valid': True}} >Thread-1965::DEBUG::2012-08-30 13:06:17,546::task::588::TaskManager.Task::(_updateState) Task=`b53d6443-efe4-4af1-9d45-50a7c90f6000`::moving from state preparing -> state finished >Thread-1965::DEBUG::2012-08-30 13:06:17,546::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1965::DEBUG::2012-08-30 13:06:17,547::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1965::DEBUG::2012-08-30 13:06:17,547::task::978::TaskManager.Task::(_decref) Task=`b53d6443-efe4-4af1-9d45-50a7c90f6000`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:06:18,462::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:20,584::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:23,463::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:23,464::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1966::DEBUG::2012-08-30 13:06:24,344::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1966::DEBUG::2012-08-30 13:06:24,344::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:06:25,585::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:06:25,847::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-1967::DEBUG::2012-08-30 13:06:26,644::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1967::DEBUG::2012-08-30 13:06:26,645::task::588::TaskManager.Task::(_updateState) Task=`f670d837-a62b-44b6-95bb-6dcfcf337953`::moving from state init -> state preparing >Thread-1967::INFO::2012-08-30 13:06:26,645::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1967::INFO::2012-08-30 13:06:26,645::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1967::DEBUG::2012-08-30 13:06:26,646::task::1172::TaskManager.Task::(prepare) Task=`f670d837-a62b-44b6-95bb-6dcfcf337953`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1967::DEBUG::2012-08-30 13:06:26,646::task::588::TaskManager.Task::(_updateState) Task=`f670d837-a62b-44b6-95bb-6dcfcf337953`::moving from state preparing -> state finished >Thread-1967::DEBUG::2012-08-30 13:06:26,646::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1967::DEBUG::2012-08-30 13:06:26,646::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1967::DEBUG::2012-08-30 13:06:26,647::task::978::TaskManager.Task::(_decref) Task=`f670d837-a62b-44b6-95bb-6dcfcf337953`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:06:28,465::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1969::DEBUG::2012-08-30 13:06:29,615::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1969::DEBUG::2012-08-30 13:06:29,616::task::588::TaskManager.Task::(_updateState) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::moving from state init -> state preparing >Thread-1969::INFO::2012-08-30 13:06:29,616::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1969::DEBUG::2012-08-30 13:06:29,617::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`eb0298bb-b2c8-4d9f-8dfd-65963dc04db1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1969::DEBUG::2012-08-30 13:06:29,617::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1969::DEBUG::2012-08-30 13:06:29,617::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1969::DEBUG::2012-08-30 13:06:29,617::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`eb0298bb-b2c8-4d9f-8dfd-65963dc04db1`::Granted request >Thread-1969::DEBUG::2012-08-30 13:06:29,618::task::817::TaskManager.Task::(resourceAcquired) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1969::DEBUG::2012-08-30 13:06:29,618::task::978::TaskManager.Task::(_decref) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::ref 1 aborting False >Thread-1969::INFO::2012-08-30 13:06:29,621::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1969::DEBUG::2012-08-30 13:06:29,621::task::1172::TaskManager.Task::(prepare) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1969::DEBUG::2012-08-30 13:06:29,621::task::588::TaskManager.Task::(_updateState) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::moving from state preparing -> state finished >Thread-1969::DEBUG::2012-08-30 13:06:29,622::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1969::DEBUG::2012-08-30 13:06:29,622::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1969::DEBUG::2012-08-30 13:06:29,622::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1969::DEBUG::2012-08-30 13:06:29,622::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1969::DEBUG::2012-08-30 13:06:29,623::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1969::DEBUG::2012-08-30 13:06:29,623::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1969::DEBUG::2012-08-30 13:06:29,623::task::978::TaskManager.Task::(_decref) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 13:06:30,586::task::588::TaskManager.Task::(_updateState) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:06:30,586::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:06:30,586::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e53ced6e-f5da-488a-b53a-a30c83435706`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:06:30,587::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:06:30,587::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:06:30,587::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e53ced6e-f5da-488a-b53a-a30c83435706`::Granted request >Thread-104::DEBUG::2012-08-30 13:06:30,587::task::817::TaskManager.Task::(resourceAcquired) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:06:30,588::task::978::TaskManager.Task::(_decref) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:06:30,589::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:06:30,591::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:06:30,592::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:06:30,593::task::1172::TaskManager.Task::(prepare) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:06:30,593::task::588::TaskManager.Task::(_updateState) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:06:30,593::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:06:30,593::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:06:30,594::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:06:30,594::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:06:30,594::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:06:30,594::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:06:30,595::task::978::TaskManager.Task::(_decref) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:06:30,595::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:30,595::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:30,596::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:30,596::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:06:32,852::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:06:33,466::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:35,597::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:38,467::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:38,467::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:40,598::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1972::DEBUG::2012-08-30 13:06:42,829::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1972::DEBUG::2012-08-30 13:06:42,830::task::588::TaskManager.Task::(_updateState) Task=`0d295874-7fc1-4786-b90e-43bdbc7e4142`::moving from state init -> state preparing >Thread-1972::INFO::2012-08-30 13:06:42,830::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1972::INFO::2012-08-30 13:06:42,830::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1972::DEBUG::2012-08-30 13:06:42,830::task::1172::TaskManager.Task::(prepare) Task=`0d295874-7fc1-4786-b90e-43bdbc7e4142`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1972::DEBUG::2012-08-30 13:06:42,831::task::588::TaskManager.Task::(_updateState) Task=`0d295874-7fc1-4786-b90e-43bdbc7e4142`::moving from state preparing -> state finished >Thread-1972::DEBUG::2012-08-30 13:06:42,831::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1972::DEBUG::2012-08-30 13:06:42,831::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1972::DEBUG::2012-08-30 13:06:42,831::task::978::TaskManager.Task::(_decref) Task=`0d295874-7fc1-4786-b90e-43bdbc7e4142`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:06:43,468::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:45,600::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:45,600::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1974::DEBUG::2012-08-30 13:06:46,149::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1974::DEBUG::2012-08-30 13:06:46,149::task::588::TaskManager.Task::(_updateState) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::moving from state init -> state preparing >Thread-1974::INFO::2012-08-30 13:06:46,149::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1974::DEBUG::2012-08-30 13:06:46,150::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9d08d300-6c0e-4079-9428-0014b2b000b9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1974::DEBUG::2012-08-30 13:06:46,150::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1974::DEBUG::2012-08-30 13:06:46,150::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1974::DEBUG::2012-08-30 13:06:46,150::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9d08d300-6c0e-4079-9428-0014b2b000b9`::Granted request >Thread-1974::DEBUG::2012-08-30 13:06:46,151::task::817::TaskManager.Task::(resourceAcquired) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1974::DEBUG::2012-08-30 13:06:46,151::task::978::TaskManager.Task::(_decref) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::ref 1 aborting False >Thread-1974::INFO::2012-08-30 13:06:46,154::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1974::DEBUG::2012-08-30 13:06:46,154::task::1172::TaskManager.Task::(prepare) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1974::DEBUG::2012-08-30 13:06:46,154::task::588::TaskManager.Task::(_updateState) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::moving from state preparing -> state finished >Thread-1974::DEBUG::2012-08-30 13:06:46,155::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1974::DEBUG::2012-08-30 13:06:46,155::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1974::DEBUG::2012-08-30 13:06:46,155::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1974::DEBUG::2012-08-30 13:06:46,155::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1974::DEBUG::2012-08-30 13:06:46,156::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1974::DEBUG::2012-08-30 13:06:46,156::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1974::DEBUG::2012-08-30 13:06:46,156::task::978::TaskManager.Task::(_decref) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:06:48,469::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1975::DEBUG::2012-08-30 13:06:49,813::task::588::TaskManager.Task::(_updateState) Task=`119154e5-5d95-40f2-9cf7-ae5fe307cd09`::moving from state init -> state preparing >Thread-1975::INFO::2012-08-30 13:06:49,813::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1975::INFO::2012-08-30 13:06:49,814::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161385536194', 'lastCheck': 1346303200.6500571, 'code': 0, 'valid': True}} >Thread-1975::DEBUG::2012-08-30 13:06:49,814::task::1172::TaskManager.Task::(prepare) Task=`119154e5-5d95-40f2-9cf7-ae5fe307cd09`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161385536194', 'lastCheck': 1346303200.6500571, 'code': 0, 'valid': True}} >Thread-1975::DEBUG::2012-08-30 13:06:49,814::task::588::TaskManager.Task::(_updateState) Task=`119154e5-5d95-40f2-9cf7-ae5fe307cd09`::moving from state preparing -> state finished >Thread-1975::DEBUG::2012-08-30 13:06:49,814::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1975::DEBUG::2012-08-30 13:06:49,815::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1975::DEBUG::2012-08-30 13:06:49,815::task::978::TaskManager.Task::(_decref) Task=`119154e5-5d95-40f2-9cf7-ae5fe307cd09`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:06:50,602::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1976::DEBUG::2012-08-30 13:06:52,997::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1976::DEBUG::2012-08-30 13:06:52,998::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::DEBUG::2012-08-30 13:06:53,470::task::588::TaskManager.Task::(_updateState) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:06:53,471::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:06:53,471::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`526dac2c-5454-4a19-866f-20d66f2f78eb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:06:53,471::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:06:53,471::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:06:53,472::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`526dac2c-5454-4a19-866f-20d66f2f78eb`::Granted request >Thread-40::DEBUG::2012-08-30 13:06:53,472::task::817::TaskManager.Task::(resourceAcquired) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:06:53,472::task::978::TaskManager.Task::(_decref) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:06:53,474::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:06:53,476::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:06:53,477::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:06:53,477::task::1172::TaskManager.Task::(prepare) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:06:53,478::task::588::TaskManager.Task::(_updateState) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:06:53,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:06:53,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:06:53,478::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:06:53,479::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:06:53,479::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:06:53,479::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:06:53,479::task::978::TaskManager.Task::(_decref) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:06:53,480::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:53,480::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:53,481::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:06:53,481::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:06:55,603::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:06:55,871::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:06:58,482::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1978::DEBUG::2012-08-30 13:06:59,352::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1978::DEBUG::2012-08-30 13:06:59,353::task::588::TaskManager.Task::(_updateState) Task=`a5c391d1-a768-445b-970e-395abf33b8f2`::moving from state init -> state preparing >Thread-1978::INFO::2012-08-30 13:06:59,353::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1978::INFO::2012-08-30 13:06:59,353::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1978::DEBUG::2012-08-30 13:06:59,353::task::1172::TaskManager.Task::(prepare) Task=`a5c391d1-a768-445b-970e-395abf33b8f2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1978::DEBUG::2012-08-30 13:06:59,354::task::588::TaskManager.Task::(_updateState) Task=`a5c391d1-a768-445b-970e-395abf33b8f2`::moving from state preparing -> state finished >Thread-1978::DEBUG::2012-08-30 13:06:59,354::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1978::DEBUG::2012-08-30 13:06:59,354::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1978::DEBUG::2012-08-30 13:06:59,354::task::978::TaskManager.Task::(_decref) Task=`a5c391d1-a768-445b-970e-395abf33b8f2`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:07:00,604::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:00,605::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1979::DEBUG::2012-08-30 13:07:02,489::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1979::DEBUG::2012-08-30 13:07:02,490::task::588::TaskManager.Task::(_updateState) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::moving from state init -> state preparing >Thread-1979::INFO::2012-08-30 13:07:02,490::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1979::DEBUG::2012-08-30 13:07:02,490::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9ebdc2cd-5a5f-4acc-a690-4031dd19d85c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1979::DEBUG::2012-08-30 13:07:02,490::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1979::DEBUG::2012-08-30 13:07:02,491::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1979::DEBUG::2012-08-30 13:07:02,491::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9ebdc2cd-5a5f-4acc-a690-4031dd19d85c`::Granted request >Thread-1979::DEBUG::2012-08-30 13:07:02,491::task::817::TaskManager.Task::(resourceAcquired) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1979::DEBUG::2012-08-30 13:07:02,492::task::978::TaskManager.Task::(_decref) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::ref 1 aborting False >Thread-1979::INFO::2012-08-30 13:07:02,495::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1979::DEBUG::2012-08-30 13:07:02,495::task::1172::TaskManager.Task::(prepare) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1979::DEBUG::2012-08-30 13:07:02,495::task::588::TaskManager.Task::(_updateState) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::moving from state preparing -> state finished >Thread-1979::DEBUG::2012-08-30 13:07:02,495::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1979::DEBUG::2012-08-30 13:07:02,495::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1979::DEBUG::2012-08-30 13:07:02,496::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1979::DEBUG::2012-08-30 13:07:02,496::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1979::DEBUG::2012-08-30 13:07:02,496::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1979::DEBUG::2012-08-30 13:07:02,496::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1979::DEBUG::2012-08-30 13:07:02,497::task::978::TaskManager.Task::(_decref) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:07:02,878::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:07:03,483::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:05,606::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:08,484::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:08,484::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:10,607::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:13,485::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:15,608::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:15,609::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1983::DEBUG::2012-08-30 13:07:15,853::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1983::DEBUG::2012-08-30 13:07:15,853::task::588::TaskManager.Task::(_updateState) Task=`2becd3dc-016e-465f-abaf-5dee715d6856`::moving from state init -> state preparing >Thread-1983::INFO::2012-08-30 13:07:15,853::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1983::INFO::2012-08-30 13:07:15,854::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1983::DEBUG::2012-08-30 13:07:15,854::task::1172::TaskManager.Task::(prepare) Task=`2becd3dc-016e-465f-abaf-5dee715d6856`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1983::DEBUG::2012-08-30 13:07:15,854::task::588::TaskManager.Task::(_updateState) Task=`2becd3dc-016e-465f-abaf-5dee715d6856`::moving from state preparing -> state finished >Thread-1983::DEBUG::2012-08-30 13:07:15,854::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1983::DEBUG::2012-08-30 13:07:15,855::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1983::DEBUG::2012-08-30 13:07:15,855::task::978::TaskManager.Task::(_decref) Task=`2becd3dc-016e-465f-abaf-5dee715d6856`::ref 0 aborting False >Thread-1984::DEBUG::2012-08-30 13:07:18,130::task::588::TaskManager.Task::(_updateState) Task=`1d048f47-6225-413e-8590-e1d890f796a5`::moving from state init -> state preparing >Thread-1984::INFO::2012-08-30 13:07:18,131::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1984::INFO::2012-08-30 13:07:18,131::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163507461548', 'lastCheck': 1346303230.670476, 'code': 0, 'valid': True}} >Thread-1984::DEBUG::2012-08-30 13:07:18,131::task::1172::TaskManager.Task::(prepare) Task=`1d048f47-6225-413e-8590-e1d890f796a5`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163507461548', 'lastCheck': 1346303230.670476, 'code': 0, 'valid': True}} >Thread-1984::DEBUG::2012-08-30 13:07:18,131::task::588::TaskManager.Task::(_updateState) Task=`1d048f47-6225-413e-8590-e1d890f796a5`::moving from state preparing -> state finished >Thread-1984::DEBUG::2012-08-30 13:07:18,132::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1984::DEBUG::2012-08-30 13:07:18,132::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1984::DEBUG::2012-08-30 13:07:18,132::task::978::TaskManager.Task::(_decref) Task=`1d048f47-6225-413e-8590-e1d890f796a5`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:07:18,486::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1985::DEBUG::2012-08-30 13:07:19,007::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1985::DEBUG::2012-08-30 13:07:19,008::task::588::TaskManager.Task::(_updateState) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::moving from state init -> state preparing >Thread-1985::INFO::2012-08-30 13:07:19,008::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1985::DEBUG::2012-08-30 13:07:19,008::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d5d377b3-0690-44fd-9e52-5a8c3b905d92`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1985::DEBUG::2012-08-30 13:07:19,008::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1985::DEBUG::2012-08-30 13:07:19,009::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1985::DEBUG::2012-08-30 13:07:19,009::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d5d377b3-0690-44fd-9e52-5a8c3b905d92`::Granted request >Thread-1985::DEBUG::2012-08-30 13:07:19,009::task::817::TaskManager.Task::(resourceAcquired) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1985::DEBUG::2012-08-30 13:07:19,009::task::978::TaskManager.Task::(_decref) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::ref 1 aborting False >Thread-1985::INFO::2012-08-30 13:07:19,012::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1985::DEBUG::2012-08-30 13:07:19,013::task::1172::TaskManager.Task::(prepare) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1985::DEBUG::2012-08-30 13:07:19,013::task::588::TaskManager.Task::(_updateState) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::moving from state preparing -> state finished >Thread-1985::DEBUG::2012-08-30 13:07:19,013::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1985::DEBUG::2012-08-30 13:07:19,013::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1985::DEBUG::2012-08-30 13:07:19,014::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1985::DEBUG::2012-08-30 13:07:19,014::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1985::DEBUG::2012-08-30 13:07:19,014::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1985::DEBUG::2012-08-30 13:07:19,014::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1985::DEBUG::2012-08-30 13:07:19,015::task::978::TaskManager.Task::(_decref) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:07:20,610::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1986::DEBUG::2012-08-30 13:07:21,294::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1986::DEBUG::2012-08-30 13:07:21,295::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:07:23,487::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:23,488::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:25,611::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:07:25,894::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:07:28,489::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:07:30,612::task::588::TaskManager.Task::(_updateState) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:07:30,613::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:07:30,613::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d504b74a-495e-4caf-acc0-8e8377277cf9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:07:30,613::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:07:30,614::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:07:30,614::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d504b74a-495e-4caf-acc0-8e8377277cf9`::Granted request >Thread-104::DEBUG::2012-08-30 13:07:30,614::task::817::TaskManager.Task::(resourceAcquired) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:07:30,615::task::978::TaskManager.Task::(_decref) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:07:30,616::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:07:30,618::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:07:30,619::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:07:30,620::task::1172::TaskManager.Task::(prepare) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:07:30,620::task::588::TaskManager.Task::(_updateState) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:07:30,620::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:07:30,620::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:07:30,620::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:07:30,621::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:07:30,621::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:07:30,621::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:07:30,621::task::978::TaskManager.Task::(_decref) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:07:30,622::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:30,622::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:30,623::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:30,623::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1989::DEBUG::2012-08-30 13:07:31,724::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1989::DEBUG::2012-08-30 13:07:31,725::task::588::TaskManager.Task::(_updateState) Task=`7e5cb8ad-88cd-4571-b6e8-5079c081cc09`::moving from state init -> state preparing >Thread-1989::INFO::2012-08-30 13:07:31,725::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1989::INFO::2012-08-30 13:07:31,726::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1989::DEBUG::2012-08-30 13:07:31,726::task::1172::TaskManager.Task::(prepare) Task=`7e5cb8ad-88cd-4571-b6e8-5079c081cc09`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1989::DEBUG::2012-08-30 13:07:31,726::task::588::TaskManager.Task::(_updateState) Task=`7e5cb8ad-88cd-4571-b6e8-5079c081cc09`::moving from state preparing -> state finished >Thread-1989::DEBUG::2012-08-30 13:07:31,726::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1989::DEBUG::2012-08-30 13:07:31,726::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1989::DEBUG::2012-08-30 13:07:31,727::task::978::TaskManager.Task::(_decref) Task=`7e5cb8ad-88cd-4571-b6e8-5079c081cc09`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:07:32,900::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:07:33,490::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1990::DEBUG::2012-08-30 13:07:34,891::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1990::DEBUG::2012-08-30 13:07:34,891::task::588::TaskManager.Task::(_updateState) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::moving from state init -> state preparing >Thread-1990::INFO::2012-08-30 13:07:34,891::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1990::DEBUG::2012-08-30 13:07:34,892::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`08b65a24-6d85-4624-a92b-95f45bf21f06`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1990::DEBUG::2012-08-30 13:07:34,892::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1990::DEBUG::2012-08-30 13:07:34,892::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1990::DEBUG::2012-08-30 13:07:34,893::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`08b65a24-6d85-4624-a92b-95f45bf21f06`::Granted request >Thread-1990::DEBUG::2012-08-30 13:07:34,893::task::817::TaskManager.Task::(resourceAcquired) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1990::DEBUG::2012-08-30 13:07:34,893::task::978::TaskManager.Task::(_decref) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::ref 1 aborting False >Thread-1990::INFO::2012-08-30 13:07:34,896::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1990::DEBUG::2012-08-30 13:07:34,896::task::1172::TaskManager.Task::(prepare) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1990::DEBUG::2012-08-30 13:07:34,897::task::588::TaskManager.Task::(_updateState) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::moving from state preparing -> state finished >Thread-1990::DEBUG::2012-08-30 13:07:34,897::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1990::DEBUG::2012-08-30 13:07:34,897::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1990::DEBUG::2012-08-30 13:07:34,897::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1990::DEBUG::2012-08-30 13:07:34,898::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1990::DEBUG::2012-08-30 13:07:34,898::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1990::DEBUG::2012-08-30 13:07:34,898::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1990::DEBUG::2012-08-30 13:07:34,898::task::978::TaskManager.Task::(_decref) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:07:35,624::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:38,491::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:38,491::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:40,625::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:43,492::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:45,627::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:45,627::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1993::DEBUG::2012-08-30 13:07:46,559::task::588::TaskManager.Task::(_updateState) Task=`d278eb17-f298-429a-abe2-05adc1b7c030`::moving from state init -> state preparing >Thread-1993::INFO::2012-08-30 13:07:46,559::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1993::INFO::2012-08-30 13:07:46,560::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346303260.6891639, 'code': 0, 'valid': True}} >Thread-1993::DEBUG::2012-08-30 13:07:46,560::task::1172::TaskManager.Task::(prepare) Task=`d278eb17-f298-429a-abe2-05adc1b7c030`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346303260.6891639, 'code': 0, 'valid': True}} >Thread-1993::DEBUG::2012-08-30 13:07:46,560::task::588::TaskManager.Task::(_updateState) Task=`d278eb17-f298-429a-abe2-05adc1b7c030`::moving from state preparing -> state finished >Thread-1993::DEBUG::2012-08-30 13:07:46,560::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1993::DEBUG::2012-08-30 13:07:46,561::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1993::DEBUG::2012-08-30 13:07:46,561::task::978::TaskManager.Task::(_decref) Task=`d278eb17-f298-429a-abe2-05adc1b7c030`::ref 0 aborting False >Thread-1994::DEBUG::2012-08-30 13:07:48,138::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1994::DEBUG::2012-08-30 13:07:48,139::task::588::TaskManager.Task::(_updateState) Task=`1ee5ed9e-3d77-4a4c-bb13-8f47ead90d7d`::moving from state init -> state preparing >Thread-1994::INFO::2012-08-30 13:07:48,139::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1994::INFO::2012-08-30 13:07:48,139::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1994::DEBUG::2012-08-30 13:07:48,140::task::1172::TaskManager.Task::(prepare) Task=`1ee5ed9e-3d77-4a4c-bb13-8f47ead90d7d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1994::DEBUG::2012-08-30 13:07:48,140::task::588::TaskManager.Task::(_updateState) Task=`1ee5ed9e-3d77-4a4c-bb13-8f47ead90d7d`::moving from state preparing -> state finished >Thread-1994::DEBUG::2012-08-30 13:07:48,140::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1994::DEBUG::2012-08-30 13:07:48,140::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1994::DEBUG::2012-08-30 13:07:48,140::task::978::TaskManager.Task::(_decref) Task=`1ee5ed9e-3d77-4a4c-bb13-8f47ead90d7d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:07:48,493::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1995::DEBUG::2012-08-30 13:07:49,715::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-1995::DEBUG::2012-08-30 13:07:49,716::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:07:50,628::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1996::DEBUG::2012-08-30 13:07:51,299::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1996::DEBUG::2012-08-30 13:07:51,299::task::588::TaskManager.Task::(_updateState) Task=`75900655-3885-48a7-9afd-379a5fd98766`::moving from state init -> state preparing >Thread-1996::INFO::2012-08-30 13:07:51,300::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1996::DEBUG::2012-08-30 13:07:51,300::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`96b96da8-9b7a-4744-bf98-69a6e041329e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1996::DEBUG::2012-08-30 13:07:51,300::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-1996::DEBUG::2012-08-30 13:07:51,301::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-1996::DEBUG::2012-08-30 13:07:51,301::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`96b96da8-9b7a-4744-bf98-69a6e041329e`::Granted request >Thread-1996::DEBUG::2012-08-30 13:07:51,301::task::817::TaskManager.Task::(resourceAcquired) Task=`75900655-3885-48a7-9afd-379a5fd98766`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-1996::DEBUG::2012-08-30 13:07:51,301::task::978::TaskManager.Task::(_decref) Task=`75900655-3885-48a7-9afd-379a5fd98766`::ref 1 aborting False >Thread-1996::INFO::2012-08-30 13:07:51,304::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1996::DEBUG::2012-08-30 13:07:51,305::task::1172::TaskManager.Task::(prepare) Task=`75900655-3885-48a7-9afd-379a5fd98766`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-1996::DEBUG::2012-08-30 13:07:51,305::task::588::TaskManager.Task::(_updateState) Task=`75900655-3885-48a7-9afd-379a5fd98766`::moving from state preparing -> state finished >Thread-1996::DEBUG::2012-08-30 13:07:51,305::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-1996::DEBUG::2012-08-30 13:07:51,305::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1996::DEBUG::2012-08-30 13:07:51,306::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-1996::DEBUG::2012-08-30 13:07:51,306::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-1996::DEBUG::2012-08-30 13:07:51,306::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-1996::DEBUG::2012-08-30 13:07:51,306::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-1996::DEBUG::2012-08-30 13:07:51,306::task::978::TaskManager.Task::(_decref) Task=`75900655-3885-48a7-9afd-379a5fd98766`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 13:07:53,494::task::588::TaskManager.Task::(_updateState) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:07:53,494::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:07:53,495::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9ecbe519-7f13-4a11-aa56-6abc51dbc34c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:07:53,495::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:07:53,495::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:07:53,495::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9ecbe519-7f13-4a11-aa56-6abc51dbc34c`::Granted request >Thread-40::DEBUG::2012-08-30 13:07:53,496::task::817::TaskManager.Task::(resourceAcquired) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:07:53,496::task::978::TaskManager.Task::(_decref) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:07:53,498::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:07:53,500::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:07:53,501::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:07:53,501::task::1172::TaskManager.Task::(prepare) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:07:53,501::task::588::TaskManager.Task::(_updateState) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:07:53,502::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:07:53,502::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:07:53,502::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:07:53,502::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:07:53,503::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:07:53,503::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:07:53,503::task::978::TaskManager.Task::(_decref) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:07:53,503::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:53,504::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:53,504::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:07:53,505::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:07:55,630::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:07:55,916::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:07:58,506::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:00,631::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:00,631::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:08:02,923::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:08:03,506::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-1999::DEBUG::2012-08-30 13:08:04,496::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-1999::DEBUG::2012-08-30 13:08:04,496::task::588::TaskManager.Task::(_updateState) Task=`2496c360-e315-4c9b-bbb3-2808fc314f1e`::moving from state init -> state preparing >Thread-1999::INFO::2012-08-30 13:08:04,496::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-1999::INFO::2012-08-30 13:08:04,497::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1999::DEBUG::2012-08-30 13:08:04,497::task::1172::TaskManager.Task::(prepare) Task=`2496c360-e315-4c9b-bbb3-2808fc314f1e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-1999::DEBUG::2012-08-30 13:08:04,497::task::588::TaskManager.Task::(_updateState) Task=`2496c360-e315-4c9b-bbb3-2808fc314f1e`::moving from state preparing -> state finished >Thread-1999::DEBUG::2012-08-30 13:08:04,497::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1999::DEBUG::2012-08-30 13:08:04,498::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1999::DEBUG::2012-08-30 13:08:04,498::task::978::TaskManager.Task::(_decref) Task=`2496c360-e315-4c9b-bbb3-2808fc314f1e`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:08:05,632::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2001::DEBUG::2012-08-30 13:08:07,762::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2001::DEBUG::2012-08-30 13:08:07,763::task::588::TaskManager.Task::(_updateState) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::moving from state init -> state preparing >Thread-2001::INFO::2012-08-30 13:08:07,763::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2001::DEBUG::2012-08-30 13:08:07,763::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`576931c6-fb6b-4341-90d2-f531ffa587b2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2001::DEBUG::2012-08-30 13:08:07,764::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2001::DEBUG::2012-08-30 13:08:07,764::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2001::DEBUG::2012-08-30 13:08:07,764::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`576931c6-fb6b-4341-90d2-f531ffa587b2`::Granted request >Thread-2001::DEBUG::2012-08-30 13:08:07,765::task::817::TaskManager.Task::(resourceAcquired) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2001::DEBUG::2012-08-30 13:08:07,765::task::978::TaskManager.Task::(_decref) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::ref 1 aborting False >Thread-2001::INFO::2012-08-30 13:08:07,768::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2001::DEBUG::2012-08-30 13:08:07,768::task::1172::TaskManager.Task::(prepare) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2001::DEBUG::2012-08-30 13:08:07,768::task::588::TaskManager.Task::(_updateState) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::moving from state preparing -> state finished >Thread-2001::DEBUG::2012-08-30 13:08:07,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2001::DEBUG::2012-08-30 13:08:07,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2001::DEBUG::2012-08-30 13:08:07,769::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2001::DEBUG::2012-08-30 13:08:07,769::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2001::DEBUG::2012-08-30 13:08:07,769::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2001::DEBUG::2012-08-30 13:08:07,770::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2001::DEBUG::2012-08-30 13:08:07,770::task::978::TaskManager.Task::(_decref) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:08:08,507::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:08,508::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:10,634::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:13,509::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2003::DEBUG::2012-08-30 13:08:15,581::task::588::TaskManager.Task::(_updateState) Task=`e5d3fce7-ee11-4192-8b8f-8796a06411b1`::moving from state init -> state preparing >Thread-2003::INFO::2012-08-30 13:08:15,581::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2003::INFO::2012-08-30 13:08:15,582::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346303290.70964, 'code': 0, 'valid': True}} >Thread-2003::DEBUG::2012-08-30 13:08:15,582::task::1172::TaskManager.Task::(prepare) Task=`e5d3fce7-ee11-4192-8b8f-8796a06411b1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346303290.70964, 'code': 0, 'valid': True}} >Thread-2003::DEBUG::2012-08-30 13:08:15,582::task::588::TaskManager.Task::(_updateState) Task=`e5d3fce7-ee11-4192-8b8f-8796a06411b1`::moving from state preparing -> state finished >Thread-2003::DEBUG::2012-08-30 13:08:15,582::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2003::DEBUG::2012-08-30 13:08:15,582::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2003::DEBUG::2012-08-30 13:08:15,583::task::978::TaskManager.Task::(_decref) Task=`e5d3fce7-ee11-4192-8b8f-8796a06411b1`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:08:15,635::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:15,636::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:18,510::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2004::DEBUG::2012-08-30 13:08:18,784::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2004::DEBUG::2012-08-30 13:08:18,784::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-2005::DEBUG::2012-08-30 13:08:20,354::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2005::DEBUG::2012-08-30 13:08:20,355::task::588::TaskManager.Task::(_updateState) Task=`f017c5d8-1305-4894-b56a-43dfd9dbede2`::moving from state init -> state preparing >Thread-2005::INFO::2012-08-30 13:08:20,355::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2005::INFO::2012-08-30 13:08:20,355::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2005::DEBUG::2012-08-30 13:08:20,355::task::1172::TaskManager.Task::(prepare) Task=`f017c5d8-1305-4894-b56a-43dfd9dbede2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2005::DEBUG::2012-08-30 13:08:20,356::task::588::TaskManager.Task::(_updateState) Task=`f017c5d8-1305-4894-b56a-43dfd9dbede2`::moving from state preparing -> state finished >Thread-2005::DEBUG::2012-08-30 13:08:20,356::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2005::DEBUG::2012-08-30 13:08:20,356::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2005::DEBUG::2012-08-30 13:08:20,356::task::978::TaskManager.Task::(_decref) Task=`f017c5d8-1305-4894-b56a-43dfd9dbede2`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:08:20,637::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:23,511::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:23,511::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2006::DEBUG::2012-08-30 13:08:23,515::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2006::DEBUG::2012-08-30 13:08:23,516::task::588::TaskManager.Task::(_updateState) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::moving from state init -> state preparing >Thread-2006::INFO::2012-08-30 13:08:23,516::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2006::DEBUG::2012-08-30 13:08:23,516::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ea018b02-bb74-423a-9cba-6fa275236f53`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2006::DEBUG::2012-08-30 13:08:23,517::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2006::DEBUG::2012-08-30 13:08:23,517::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2006::DEBUG::2012-08-30 13:08:23,518::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ea018b02-bb74-423a-9cba-6fa275236f53`::Granted request >Thread-2006::DEBUG::2012-08-30 13:08:23,518::task::817::TaskManager.Task::(resourceAcquired) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2006::DEBUG::2012-08-30 13:08:23,518::task::978::TaskManager.Task::(_decref) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::ref 1 aborting False >Thread-2006::INFO::2012-08-30 13:08:23,521::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2006::DEBUG::2012-08-30 13:08:23,521::task::1172::TaskManager.Task::(prepare) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2006::DEBUG::2012-08-30 13:08:23,522::task::588::TaskManager.Task::(_updateState) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::moving from state preparing -> state finished >Thread-2006::DEBUG::2012-08-30 13:08:23,522::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2006::DEBUG::2012-08-30 13:08:23,522::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2006::DEBUG::2012-08-30 13:08:23,522::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2006::DEBUG::2012-08-30 13:08:23,523::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2006::DEBUG::2012-08-30 13:08:23,523::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2006::DEBUG::2012-08-30 13:08:23,523::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2006::DEBUG::2012-08-30 13:08:23,523::task::978::TaskManager.Task::(_decref) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:08:25,638::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:08:25,943::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:08:28,512::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:08:30,639::task::588::TaskManager.Task::(_updateState) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:08:30,640::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:08:30,640::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f08f061c-8b62-403e-a48e-8edbbc716fe5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:08:30,640::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:08:30,640::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:08:30,641::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f08f061c-8b62-403e-a48e-8edbbc716fe5`::Granted request >Thread-104::DEBUG::2012-08-30 13:08:30,641::task::817::TaskManager.Task::(resourceAcquired) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:08:30,641::task::978::TaskManager.Task::(_decref) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:08:30,643::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:08:30,645::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:08:30,646::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:08:30,646::task::1172::TaskManager.Task::(prepare) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:08:30,646::task::588::TaskManager.Task::(_updateState) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:08:30,647::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:08:30,647::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:08:30,647::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:08:30,647::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:08:30,648::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:08:30,648::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:08:30,648::task::978::TaskManager.Task::(_decref) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:08:30,649::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:30,649::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:30,649::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:30,650::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:08:32,947::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:08:33,513::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:35,651::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2010::DEBUG::2012-08-30 13:08:36,698::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2010::DEBUG::2012-08-30 13:08:36,699::task::588::TaskManager.Task::(_updateState) Task=`679a1e33-2ad5-4e72-bbd0-1349e6e4dbc6`::moving from state init -> state preparing >Thread-2010::INFO::2012-08-30 13:08:36,699::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2010::INFO::2012-08-30 13:08:36,699::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2010::DEBUG::2012-08-30 13:08:36,699::task::1172::TaskManager.Task::(prepare) Task=`679a1e33-2ad5-4e72-bbd0-1349e6e4dbc6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2010::DEBUG::2012-08-30 13:08:36,700::task::588::TaskManager.Task::(_updateState) Task=`679a1e33-2ad5-4e72-bbd0-1349e6e4dbc6`::moving from state preparing -> state finished >Thread-2010::DEBUG::2012-08-30 13:08:36,700::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2010::DEBUG::2012-08-30 13:08:36,700::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2010::DEBUG::2012-08-30 13:08:36,700::task::978::TaskManager.Task::(_decref) Task=`679a1e33-2ad5-4e72-bbd0-1349e6e4dbc6`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:08:38,514::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:38,515::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2012::DEBUG::2012-08-30 13:08:39,842::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2012::DEBUG::2012-08-30 13:08:39,843::task::588::TaskManager.Task::(_updateState) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::moving from state init -> state preparing >Thread-2012::INFO::2012-08-30 13:08:39,843::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2012::DEBUG::2012-08-30 13:08:39,843::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`362c63b4-fbca-4040-b737-e7432b5bf585`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2012::DEBUG::2012-08-30 13:08:39,843::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2012::DEBUG::2012-08-30 13:08:39,844::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2012::DEBUG::2012-08-30 13:08:39,844::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`362c63b4-fbca-4040-b737-e7432b5bf585`::Granted request >Thread-2012::DEBUG::2012-08-30 13:08:39,844::task::817::TaskManager.Task::(resourceAcquired) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2012::DEBUG::2012-08-30 13:08:39,844::task::978::TaskManager.Task::(_decref) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::ref 1 aborting False >Thread-2012::INFO::2012-08-30 13:08:39,847::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2012::DEBUG::2012-08-30 13:08:39,848::task::1172::TaskManager.Task::(prepare) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2012::DEBUG::2012-08-30 13:08:39,848::task::588::TaskManager.Task::(_updateState) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::moving from state preparing -> state finished >Thread-2012::DEBUG::2012-08-30 13:08:39,848::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2012::DEBUG::2012-08-30 13:08:39,848::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2012::DEBUG::2012-08-30 13:08:39,849::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2012::DEBUG::2012-08-30 13:08:39,849::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2012::DEBUG::2012-08-30 13:08:39,849::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2012::DEBUG::2012-08-30 13:08:39,849::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2012::DEBUG::2012-08-30 13:08:39,850::task::978::TaskManager.Task::(_decref) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:08:40,652::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:43,515::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2013::DEBUG::2012-08-30 13:08:44,154::task::588::TaskManager.Task::(_updateState) Task=`291e2646-a082-4b65-b3ac-528435a71888`::moving from state init -> state preparing >Thread-2013::INFO::2012-08-30 13:08:44,154::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2013::INFO::2012-08-30 13:08:44,155::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163006782532', 'lastCheck': 1346303320.728425, 'code': 0, 'valid': True}} >Thread-2013::DEBUG::2012-08-30 13:08:44,155::task::1172::TaskManager.Task::(prepare) Task=`291e2646-a082-4b65-b3ac-528435a71888`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163006782532', 'lastCheck': 1346303320.728425, 'code': 0, 'valid': True}} >Thread-2013::DEBUG::2012-08-30 13:08:44,155::task::588::TaskManager.Task::(_updateState) Task=`291e2646-a082-4b65-b3ac-528435a71888`::moving from state preparing -> state finished >Thread-2013::DEBUG::2012-08-30 13:08:44,155::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2013::DEBUG::2012-08-30 13:08:44,156::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2013::DEBUG::2012-08-30 13:08:44,156::task::978::TaskManager.Task::(_decref) Task=`291e2646-a082-4b65-b3ac-528435a71888`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:08:45,654::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:45,654::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2014::DEBUG::2012-08-30 13:08:47,335::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2014::DEBUG::2012-08-30 13:08:47,336::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:08:48,516::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:50,655::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2016::DEBUG::2012-08-30 13:08:52,934::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2016::DEBUG::2012-08-30 13:08:52,934::task::588::TaskManager.Task::(_updateState) Task=`5fa5fe1f-df7b-419d-a520-3234c010315c`::moving from state init -> state preparing >Thread-2016::INFO::2012-08-30 13:08:52,935::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2016::INFO::2012-08-30 13:08:52,935::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2016::DEBUG::2012-08-30 13:08:52,935::task::1172::TaskManager.Task::(prepare) Task=`5fa5fe1f-df7b-419d-a520-3234c010315c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2016::DEBUG::2012-08-30 13:08:52,936::task::588::TaskManager.Task::(_updateState) Task=`5fa5fe1f-df7b-419d-a520-3234c010315c`::moving from state preparing -> state finished >Thread-2016::DEBUG::2012-08-30 13:08:52,936::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2016::DEBUG::2012-08-30 13:08:52,936::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2016::DEBUG::2012-08-30 13:08:52,936::task::978::TaskManager.Task::(_decref) Task=`5fa5fe1f-df7b-419d-a520-3234c010315c`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 13:08:53,517::task::588::TaskManager.Task::(_updateState) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:08:53,518::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:08:53,518::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`b13fd91b-e2d2-4422-a0c0-d0908adfddb6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:08:53,518::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:08:53,518::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:08:53,519::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`b13fd91b-e2d2-4422-a0c0-d0908adfddb6`::Granted request >Thread-40::DEBUG::2012-08-30 13:08:53,519::task::817::TaskManager.Task::(resourceAcquired) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:08:53,519::task::978::TaskManager.Task::(_decref) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:08:53,521::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:08:53,523::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:08:53,524::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:08:53,524::task::1172::TaskManager.Task::(prepare) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:08:53,525::task::588::TaskManager.Task::(_updateState) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:08:53,525::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:08:53,525::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:08:53,525::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:08:53,526::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:08:53,526::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:08:53,526::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:08:53,526::task::978::TaskManager.Task::(_decref) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:08:53,527::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:53,527::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:53,528::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:08:53,528::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:08:55,657::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:08:55,966::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2017::DEBUG::2012-08-30 13:08:56,084::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2017::DEBUG::2012-08-30 13:08:56,085::task::588::TaskManager.Task::(_updateState) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::moving from state init -> state preparing >Thread-2017::INFO::2012-08-30 13:08:56,085::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2017::DEBUG::2012-08-30 13:08:56,085::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`975c8ec8-c6ce-410d-a9ca-418af24c6484`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2017::DEBUG::2012-08-30 13:08:56,086::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2017::DEBUG::2012-08-30 13:08:56,086::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2017::DEBUG::2012-08-30 13:08:56,086::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`975c8ec8-c6ce-410d-a9ca-418af24c6484`::Granted request >Thread-2017::DEBUG::2012-08-30 13:08:56,086::task::817::TaskManager.Task::(resourceAcquired) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2017::DEBUG::2012-08-30 13:08:56,087::task::978::TaskManager.Task::(_decref) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::ref 1 aborting False >Thread-2017::INFO::2012-08-30 13:08:56,090::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2017::DEBUG::2012-08-30 13:08:56,090::task::1172::TaskManager.Task::(prepare) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2017::DEBUG::2012-08-30 13:08:56,090::task::588::TaskManager.Task::(_updateState) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::moving from state preparing -> state finished >Thread-2017::DEBUG::2012-08-30 13:08:56,090::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2017::DEBUG::2012-08-30 13:08:56,091::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2017::DEBUG::2012-08-30 13:08:56,091::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2017::DEBUG::2012-08-30 13:08:56,091::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2017::DEBUG::2012-08-30 13:08:56,091::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2017::DEBUG::2012-08-30 13:08:56,092::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2017::DEBUG::2012-08-30 13:08:56,092::task::978::TaskManager.Task::(_decref) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:08:58,529::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:00,658::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:00,658::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:09:02,973::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:09:03,530::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:05,659::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:08,531::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:08,531::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2021::DEBUG::2012-08-30 13:09:09,477::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2021::DEBUG::2012-08-30 13:09:09,477::task::588::TaskManager.Task::(_updateState) Task=`42319c1e-eb08-479a-8ee9-57703ffcd3b5`::moving from state init -> state preparing >Thread-2021::INFO::2012-08-30 13:09:09,477::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2021::INFO::2012-08-30 13:09:09,478::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2021::DEBUG::2012-08-30 13:09:09,478::task::1172::TaskManager.Task::(prepare) Task=`42319c1e-eb08-479a-8ee9-57703ffcd3b5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2021::DEBUG::2012-08-30 13:09:09,478::task::588::TaskManager.Task::(_updateState) Task=`42319c1e-eb08-479a-8ee9-57703ffcd3b5`::moving from state preparing -> state finished >Thread-2021::DEBUG::2012-08-30 13:09:09,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2021::DEBUG::2012-08-30 13:09:09,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2021::DEBUG::2012-08-30 13:09:09,479::task::978::TaskManager.Task::(_decref) Task=`42319c1e-eb08-479a-8ee9-57703ffcd3b5`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:09:10,660::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2022::DEBUG::2012-08-30 13:09:12,895::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2023::DEBUG::2012-08-30 13:09:12,896::task::588::TaskManager.Task::(_updateState) Task=`73d3a043-0fe9-4160-9539-44ffced6508f`::moving from state init -> state preparing >Thread-2022::DEBUG::2012-08-30 13:09:12,896::task::588::TaskManager.Task::(_updateState) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::moving from state init -> state preparing >Thread-2023::INFO::2012-08-30 13:09:12,896::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2022::INFO::2012-08-30 13:09:12,896::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2023::INFO::2012-08-30 13:09:12,897::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164389610291', 'lastCheck': 1346303350.7488611, 'code': 0, 'valid': True}} >Thread-2022::DEBUG::2012-08-30 13:09:12,897::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`06036aad-493a-4d68-9987-2efbdbb4c731`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2023::DEBUG::2012-08-30 13:09:12,897::task::1172::TaskManager.Task::(prepare) Task=`73d3a043-0fe9-4160-9539-44ffced6508f`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164389610291', 'lastCheck': 1346303350.7488611, 'code': 0, 'valid': True}} >Thread-2022::DEBUG::2012-08-30 13:09:12,897::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2023::DEBUG::2012-08-30 13:09:12,898::task::588::TaskManager.Task::(_updateState) Task=`73d3a043-0fe9-4160-9539-44ffced6508f`::moving from state preparing -> state finished >Thread-2022::DEBUG::2012-08-30 13:09:12,898::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2023::DEBUG::2012-08-30 13:09:12,898::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2022::DEBUG::2012-08-30 13:09:12,898::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`06036aad-493a-4d68-9987-2efbdbb4c731`::Granted request >Thread-2023::DEBUG::2012-08-30 13:09:12,898::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2023::DEBUG::2012-08-30 13:09:12,899::task::978::TaskManager.Task::(_decref) Task=`73d3a043-0fe9-4160-9539-44ffced6508f`::ref 0 aborting False >Thread-2022::DEBUG::2012-08-30 13:09:12,900::task::817::TaskManager.Task::(resourceAcquired) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2022::DEBUG::2012-08-30 13:09:12,900::task::978::TaskManager.Task::(_decref) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::ref 1 aborting False >Thread-2022::INFO::2012-08-30 13:09:12,905::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2022::DEBUG::2012-08-30 13:09:12,905::task::1172::TaskManager.Task::(prepare) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2022::DEBUG::2012-08-30 13:09:12,905::task::588::TaskManager.Task::(_updateState) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::moving from state preparing -> state finished >Thread-2022::DEBUG::2012-08-30 13:09:12,905::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2022::DEBUG::2012-08-30 13:09:12,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2022::DEBUG::2012-08-30 13:09:12,906::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2022::DEBUG::2012-08-30 13:09:12,906::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2022::DEBUG::2012-08-30 13:09:12,907::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2022::DEBUG::2012-08-30 13:09:12,907::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2022::DEBUG::2012-08-30 13:09:12,907::task::978::TaskManager.Task::(_decref) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:09:13,532::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:15,662::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:15,662::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2024::DEBUG::2012-08-30 13:09:16,075::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2024::DEBUG::2012-08-30 13:09:16,075::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:09:18,533::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:20,663::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:23,534::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:23,535::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:25,665::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:09:25,993::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2026::DEBUG::2012-08-30 13:09:26,178::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2026::DEBUG::2012-08-30 13:09:26,178::task::588::TaskManager.Task::(_updateState) Task=`1d31158c-78f0-4688-9576-0e703051b93b`::moving from state init -> state preparing >Thread-2026::INFO::2012-08-30 13:09:26,178::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2026::INFO::2012-08-30 13:09:26,179::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2026::DEBUG::2012-08-30 13:09:26,179::task::1172::TaskManager.Task::(prepare) Task=`1d31158c-78f0-4688-9576-0e703051b93b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2026::DEBUG::2012-08-30 13:09:26,179::task::588::TaskManager.Task::(_updateState) Task=`1d31158c-78f0-4688-9576-0e703051b93b`::moving from state preparing -> state finished >Thread-2026::DEBUG::2012-08-30 13:09:26,180::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2026::DEBUG::2012-08-30 13:09:26,180::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2026::DEBUG::2012-08-30 13:09:26,180::task::978::TaskManager.Task::(_decref) Task=`1d31158c-78f0-4688-9576-0e703051b93b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:09:28,536::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2028::DEBUG::2012-08-30 13:09:29,413::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2028::DEBUG::2012-08-30 13:09:29,414::task::588::TaskManager.Task::(_updateState) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::moving from state init -> state preparing >Thread-2028::INFO::2012-08-30 13:09:29,414::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2028::DEBUG::2012-08-30 13:09:29,414::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`78309f2c-f36e-4514-8735-d03ebec19a64`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2028::DEBUG::2012-08-30 13:09:29,414::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2028::DEBUG::2012-08-30 13:09:29,415::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2028::DEBUG::2012-08-30 13:09:29,415::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`78309f2c-f36e-4514-8735-d03ebec19a64`::Granted request >Thread-2028::DEBUG::2012-08-30 13:09:29,415::task::817::TaskManager.Task::(resourceAcquired) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2028::DEBUG::2012-08-30 13:09:29,416::task::978::TaskManager.Task::(_decref) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::ref 1 aborting False >Thread-2028::INFO::2012-08-30 13:09:29,418::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2028::DEBUG::2012-08-30 13:09:29,419::task::1172::TaskManager.Task::(prepare) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2028::DEBUG::2012-08-30 13:09:29,419::task::588::TaskManager.Task::(_updateState) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::moving from state preparing -> state finished >Thread-2028::DEBUG::2012-08-30 13:09:29,419::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2028::DEBUG::2012-08-30 13:09:29,419::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2028::DEBUG::2012-08-30 13:09:29,420::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2028::DEBUG::2012-08-30 13:09:29,420::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2028::DEBUG::2012-08-30 13:09:29,420::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2028::DEBUG::2012-08-30 13:09:29,420::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2028::DEBUG::2012-08-30 13:09:29,421::task::978::TaskManager.Task::(_decref) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 13:09:30,666::task::588::TaskManager.Task::(_updateState) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:09:30,666::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:09:30,666::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`3f8eb9ad-1d2a-46ea-aeb6-fbdace8cc8f1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:09:30,667::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:09:30,667::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:09:30,667::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`3f8eb9ad-1d2a-46ea-aeb6-fbdace8cc8f1`::Granted request >Thread-104::DEBUG::2012-08-30 13:09:30,667::task::817::TaskManager.Task::(resourceAcquired) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:09:30,668::task::978::TaskManager.Task::(_decref) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:09:30,669::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:09:30,671::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:09:30,672::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:09:30,673::task::1172::TaskManager.Task::(prepare) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:09:30,673::task::588::TaskManager.Task::(_updateState) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:09:30,673::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:09:30,673::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:09:30,674::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:09:30,674::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:09:30,674::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:09:30,674::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:09:30,675::task::978::TaskManager.Task::(_decref) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:09:30,675::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:30,675::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:30,676::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:30,676::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:09:33,000::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:09:33,537::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:35,677::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:38,537::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:38,538::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:40,678::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2031::DEBUG::2012-08-30 13:09:41,596::task::588::TaskManager.Task::(_updateState) Task=`6ccf8132-21e3-4298-ad0a-9b9ec81e2654`::moving from state init -> state preparing >Thread-2031::INFO::2012-08-30 13:09:41,597::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2031::INFO::2012-08-30 13:09:41,597::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161385536194', 'lastCheck': 1346303380.7674611, 'code': 0, 'valid': True}} >Thread-2031::DEBUG::2012-08-30 13:09:41,597::task::1172::TaskManager.Task::(prepare) Task=`6ccf8132-21e3-4298-ad0a-9b9ec81e2654`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161385536194', 'lastCheck': 1346303380.7674611, 'code': 0, 'valid': True}} >Thread-2031::DEBUG::2012-08-30 13:09:41,597::task::588::TaskManager.Task::(_updateState) Task=`6ccf8132-21e3-4298-ad0a-9b9ec81e2654`::moving from state preparing -> state finished >Thread-2031::DEBUG::2012-08-30 13:09:41,597::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2031::DEBUG::2012-08-30 13:09:41,598::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2031::DEBUG::2012-08-30 13:09:41,598::task::978::TaskManager.Task::(_decref) Task=`6ccf8132-21e3-4298-ad0a-9b9ec81e2654`::ref 0 aborting False >Thread-2032::DEBUG::2012-08-30 13:09:43,163::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2032::DEBUG::2012-08-30 13:09:43,164::task::588::TaskManager.Task::(_updateState) Task=`4d377e3d-3d81-4597-93ac-41ee1bd43042`::moving from state init -> state preparing >Thread-2032::INFO::2012-08-30 13:09:43,164::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2032::INFO::2012-08-30 13:09:43,164::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2032::DEBUG::2012-08-30 13:09:43,164::task::1172::TaskManager.Task::(prepare) Task=`4d377e3d-3d81-4597-93ac-41ee1bd43042`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2032::DEBUG::2012-08-30 13:09:43,165::task::588::TaskManager.Task::(_updateState) Task=`4d377e3d-3d81-4597-93ac-41ee1bd43042`::moving from state preparing -> state finished >Thread-2032::DEBUG::2012-08-30 13:09:43,165::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2032::DEBUG::2012-08-30 13:09:43,165::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2032::DEBUG::2012-08-30 13:09:43,165::task::978::TaskManager.Task::(_decref) Task=`4d377e3d-3d81-4597-93ac-41ee1bd43042`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:09:43,539::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2033::DEBUG::2012-08-30 13:09:44,723::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2033::DEBUG::2012-08-30 13:09:44,723::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:09:45,680::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:45,680::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2034::DEBUG::2012-08-30 13:09:46,289::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2034::DEBUG::2012-08-30 13:09:46,289::task::588::TaskManager.Task::(_updateState) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::moving from state init -> state preparing >Thread-2034::INFO::2012-08-30 13:09:46,289::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2034::DEBUG::2012-08-30 13:09:46,290::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`173bf5dd-0a3a-4e4c-8d6d-d464e496c174`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2034::DEBUG::2012-08-30 13:09:46,290::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2034::DEBUG::2012-08-30 13:09:46,290::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2034::DEBUG::2012-08-30 13:09:46,290::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`173bf5dd-0a3a-4e4c-8d6d-d464e496c174`::Granted request >Thread-2034::DEBUG::2012-08-30 13:09:46,291::task::817::TaskManager.Task::(resourceAcquired) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2034::DEBUG::2012-08-30 13:09:46,291::task::978::TaskManager.Task::(_decref) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::ref 1 aborting False >Thread-2034::INFO::2012-08-30 13:09:46,294::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2034::DEBUG::2012-08-30 13:09:46,294::task::1172::TaskManager.Task::(prepare) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2034::DEBUG::2012-08-30 13:09:46,294::task::588::TaskManager.Task::(_updateState) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::moving from state preparing -> state finished >Thread-2034::DEBUG::2012-08-30 13:09:46,295::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2034::DEBUG::2012-08-30 13:09:46,295::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2034::DEBUG::2012-08-30 13:09:46,295::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2034::DEBUG::2012-08-30 13:09:46,295::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2034::DEBUG::2012-08-30 13:09:46,296::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2034::DEBUG::2012-08-30 13:09:46,296::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2034::DEBUG::2012-08-30 13:09:46,296::task::978::TaskManager.Task::(_decref) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:09:48,540::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:50,682::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:09:53,541::task::588::TaskManager.Task::(_updateState) Task=`be20643a-e938-4378-982a-0fe69b075d56`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:09:53,541::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:09:53,542::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`333a414e-7712-4e24-a688-88147fe52e1d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:09:53,542::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:09:53,542::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:09:53,542::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`333a414e-7712-4e24-a688-88147fe52e1d`::Granted request >Thread-40::DEBUG::2012-08-30 13:09:53,543::task::817::TaskManager.Task::(resourceAcquired) Task=`be20643a-e938-4378-982a-0fe69b075d56`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:09:53,543::task::978::TaskManager.Task::(_decref) Task=`be20643a-e938-4378-982a-0fe69b075d56`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:09:53,545::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:09:53,547::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:09:53,548::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:09:53,548::task::1172::TaskManager.Task::(prepare) Task=`be20643a-e938-4378-982a-0fe69b075d56`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:09:53,548::task::588::TaskManager.Task::(_updateState) Task=`be20643a-e938-4378-982a-0fe69b075d56`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:09:53,549::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:09:53,549::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:09:53,549::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:09:53,549::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:09:53,550::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:09:53,550::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:09:53,550::task::978::TaskManager.Task::(_decref) Task=`be20643a-e938-4378-982a-0fe69b075d56`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:09:53,550::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:53,551::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:53,551::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:09:53,552::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:09:55,683::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:09:56,023::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:09:58,553::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2037::DEBUG::2012-08-30 13:09:59,509::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2037::DEBUG::2012-08-30 13:09:59,509::task::588::TaskManager.Task::(_updateState) Task=`0567c659-5b94-412c-8282-69f9d03c88b3`::moving from state init -> state preparing >Thread-2037::INFO::2012-08-30 13:09:59,509::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2037::INFO::2012-08-30 13:09:59,510::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2037::DEBUG::2012-08-30 13:09:59,510::task::1172::TaskManager.Task::(prepare) Task=`0567c659-5b94-412c-8282-69f9d03c88b3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2037::DEBUG::2012-08-30 13:09:59,510::task::588::TaskManager.Task::(_updateState) Task=`0567c659-5b94-412c-8282-69f9d03c88b3`::moving from state preparing -> state finished >Thread-2037::DEBUG::2012-08-30 13:09:59,510::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2037::DEBUG::2012-08-30 13:09:59,511::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2037::DEBUG::2012-08-30 13:09:59,511::task::978::TaskManager.Task::(_decref) Task=`0567c659-5b94-412c-8282-69f9d03c88b3`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:10:00,684::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:00,685::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2039::DEBUG::2012-08-30 13:10:02,652::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2039::DEBUG::2012-08-30 13:10:02,652::task::588::TaskManager.Task::(_updateState) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::moving from state init -> state preparing >Thread-2039::INFO::2012-08-30 13:10:02,653::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2039::DEBUG::2012-08-30 13:10:02,653::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`51048185-3e73-475f-a68f-c733d8b2086e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2039::DEBUG::2012-08-30 13:10:02,653::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2039::DEBUG::2012-08-30 13:10:02,654::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2039::DEBUG::2012-08-30 13:10:02,654::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`51048185-3e73-475f-a68f-c733d8b2086e`::Granted request >Thread-2039::DEBUG::2012-08-30 13:10:02,654::task::817::TaskManager.Task::(resourceAcquired) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2039::DEBUG::2012-08-30 13:10:02,654::task::978::TaskManager.Task::(_decref) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::ref 1 aborting False >Thread-2039::INFO::2012-08-30 13:10:02,658::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2039::DEBUG::2012-08-30 13:10:02,658::task::1172::TaskManager.Task::(prepare) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2039::DEBUG::2012-08-30 13:10:02,658::task::588::TaskManager.Task::(_updateState) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::moving from state preparing -> state finished >Thread-2039::DEBUG::2012-08-30 13:10:02,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2039::DEBUG::2012-08-30 13:10:02,659::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2039::DEBUG::2012-08-30 13:10:02,659::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2039::DEBUG::2012-08-30 13:10:02,659::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2039::DEBUG::2012-08-30 13:10:02,659::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2039::DEBUG::2012-08-30 13:10:02,660::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2039::DEBUG::2012-08-30 13:10:02,660::task::978::TaskManager.Task::(_decref) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:10:03,030::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:10:03,554::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:05,686::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:08,555::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:08,555::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2041::DEBUG::2012-08-30 13:10:10,615::task::588::TaskManager.Task::(_updateState) Task=`7c96cc23-a5c0-4e7b-b1eb-a6206a156b6c`::moving from state init -> state preparing >Thread-2041::INFO::2012-08-30 13:10:10,615::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2041::INFO::2012-08-30 13:10:10,616::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177717208862', 'lastCheck': 1346303400.7816091, 'code': 0, 'valid': True}} >Thread-2041::DEBUG::2012-08-30 13:10:10,616::task::1172::TaskManager.Task::(prepare) Task=`7c96cc23-a5c0-4e7b-b1eb-a6206a156b6c`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177717208862', 'lastCheck': 1346303400.7816091, 'code': 0, 'valid': True}} >Thread-2041::DEBUG::2012-08-30 13:10:10,616::task::588::TaskManager.Task::(_updateState) Task=`7c96cc23-a5c0-4e7b-b1eb-a6206a156b6c`::moving from state preparing -> state finished >Thread-2041::DEBUG::2012-08-30 13:10:10,616::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2041::DEBUG::2012-08-30 13:10:10,617::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2041::DEBUG::2012-08-30 13:10:10,617::task::978::TaskManager.Task::(_decref) Task=`7c96cc23-a5c0-4e7b-b1eb-a6206a156b6c`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:10:10,686::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:13,556::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2042::DEBUG::2012-08-30 13:10:13,782::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2042::DEBUG::2012-08-30 13:10:13,782::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-2043::DEBUG::2012-08-30 13:10:15,348::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2043::DEBUG::2012-08-30 13:10:15,349::task::588::TaskManager.Task::(_updateState) Task=`b2cf501b-0753-44bc-9ce1-c8518da0a80b`::moving from state init -> state preparing >Thread-2043::INFO::2012-08-30 13:10:15,349::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2043::INFO::2012-08-30 13:10:15,349::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2043::DEBUG::2012-08-30 13:10:15,350::task::1172::TaskManager.Task::(prepare) Task=`b2cf501b-0753-44bc-9ce1-c8518da0a80b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2043::DEBUG::2012-08-30 13:10:15,350::task::588::TaskManager.Task::(_updateState) Task=`b2cf501b-0753-44bc-9ce1-c8518da0a80b`::moving from state preparing -> state finished >Thread-2043::DEBUG::2012-08-30 13:10:15,350::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2043::DEBUG::2012-08-30 13:10:15,350::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2043::DEBUG::2012-08-30 13:10:15,351::task::978::TaskManager.Task::(_decref) Task=`b2cf501b-0753-44bc-9ce1-c8518da0a80b`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:10:15,687::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:15,688::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2044::DEBUG::2012-08-30 13:10:18,512::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2044::DEBUG::2012-08-30 13:10:18,512::task::588::TaskManager.Task::(_updateState) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::moving from state init -> state preparing >Thread-2044::INFO::2012-08-30 13:10:18,512::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2044::DEBUG::2012-08-30 13:10:18,513::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`888e3066-7967-48e0-8819-71dc23bb8e4d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2044::DEBUG::2012-08-30 13:10:18,513::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2044::DEBUG::2012-08-30 13:10:18,513::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2044::DEBUG::2012-08-30 13:10:18,514::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`888e3066-7967-48e0-8819-71dc23bb8e4d`::Granted request >Thread-2044::DEBUG::2012-08-30 13:10:18,514::task::817::TaskManager.Task::(resourceAcquired) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2044::DEBUG::2012-08-30 13:10:18,514::task::978::TaskManager.Task::(_decref) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::ref 1 aborting False >Thread-2044::INFO::2012-08-30 13:10:18,518::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2044::DEBUG::2012-08-30 13:10:18,518::task::1172::TaskManager.Task::(prepare) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2044::DEBUG::2012-08-30 13:10:18,518::task::588::TaskManager.Task::(_updateState) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::moving from state preparing -> state finished >Thread-2044::DEBUG::2012-08-30 13:10:18,519::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2044::DEBUG::2012-08-30 13:10:18,519::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2044::DEBUG::2012-08-30 13:10:18,519::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2044::DEBUG::2012-08-30 13:10:18,519::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2044::DEBUG::2012-08-30 13:10:18,520::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2044::DEBUG::2012-08-30 13:10:18,520::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2044::DEBUG::2012-08-30 13:10:18,520::task::978::TaskManager.Task::(_decref) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:10:18,558::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:20,689::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:23,559::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:23,560::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:25,690::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:10:26,051::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:10:28,561::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:10:30,691::task::588::TaskManager.Task::(_updateState) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:10:30,691::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:10:30,691::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`33ba124f-5630-4922-87b3-90bf952b0e34`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:10:30,692::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:10:30,692::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:10:30,692::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`33ba124f-5630-4922-87b3-90bf952b0e34`::Granted request >Thread-104::DEBUG::2012-08-30 13:10:30,692::task::817::TaskManager.Task::(resourceAcquired) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:10:30,693::task::978::TaskManager.Task::(_decref) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:10:30,694::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:10:30,696::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:10:30,697::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:10:30,698::task::1172::TaskManager.Task::(prepare) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:10:30,698::task::588::TaskManager.Task::(_updateState) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:10:30,698::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:10:30,698::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:10:30,699::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:10:30,699::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:10:30,699::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:10:30,699::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:10:30,700::task::978::TaskManager.Task::(_decref) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:10:30,700::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:30,700::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:30,701::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:30,701::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-26::DEBUG::2012-08-30 13:10:30,794::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:10:30,794::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-26::DEBUG::2012-08-30 13:10:30,795::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None) >Thread-26::DEBUG::2012-08-30 13:10:30,905::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5 >Thread-26::WARNING::2012-08-30 13:10:30,906::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found'] >Thread-26::DEBUG::2012-08-30 13:10:30,906::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-26::DEBUG::2012-08-30 13:10:30,912::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:10:30,913::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend >Thread-26::DEBUG::2012-08-30 13:10:30,915::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d'] >Thread-26::DEBUG::2012-08-30 13:10:30,916::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-26::WARNING::2012-08-30 13:10:30,916::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered >Thread-26::WARNING::2012-08-30 13:10:30,917::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered >Thread-2048::DEBUG::2012-08-30 13:10:32,155::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2048::DEBUG::2012-08-30 13:10:32,155::task::588::TaskManager.Task::(_updateState) Task=`1e524eb9-c3a1-4a7b-b1d3-d7719800d9cf`::moving from state init -> state preparing >Thread-2048::INFO::2012-08-30 13:10:32,155::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2048::INFO::2012-08-30 13:10:32,156::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2048::DEBUG::2012-08-30 13:10:32,156::task::1172::TaskManager.Task::(prepare) Task=`1e524eb9-c3a1-4a7b-b1d3-d7719800d9cf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2048::DEBUG::2012-08-30 13:10:32,156::task::588::TaskManager.Task::(_updateState) Task=`1e524eb9-c3a1-4a7b-b1d3-d7719800d9cf`::moving from state preparing -> state finished >Thread-2048::DEBUG::2012-08-30 13:10:32,156::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2048::DEBUG::2012-08-30 13:10:32,157::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2048::DEBUG::2012-08-30 13:10:32,157::task::978::TaskManager.Task::(_decref) Task=`1e524eb9-c3a1-4a7b-b1d3-d7719800d9cf`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:10:33,058::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:10:33,563::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:35,702::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2055::DEBUG::2012-08-30 13:10:36,274::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2055::DEBUG::2012-08-30 13:10:36,275::task::588::TaskManager.Task::(_updateState) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::moving from state init -> state preparing >Thread-2055::INFO::2012-08-30 13:10:36,275::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2055::DEBUG::2012-08-30 13:10:36,275::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`37a0c218-4f6e-49f2-92e0-42bcd3b5fb4a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2055::DEBUG::2012-08-30 13:10:36,276::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2055::DEBUG::2012-08-30 13:10:36,276::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2055::DEBUG::2012-08-30 13:10:36,276::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`37a0c218-4f6e-49f2-92e0-42bcd3b5fb4a`::Granted request >Thread-2055::DEBUG::2012-08-30 13:10:36,276::task::817::TaskManager.Task::(resourceAcquired) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2055::DEBUG::2012-08-30 13:10:36,277::task::978::TaskManager.Task::(_decref) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::ref 1 aborting False >Thread-2055::INFO::2012-08-30 13:10:36,280::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2055::DEBUG::2012-08-30 13:10:36,280::task::1172::TaskManager.Task::(prepare) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2055::DEBUG::2012-08-30 13:10:36,280::task::588::TaskManager.Task::(_updateState) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::moving from state preparing -> state finished >Thread-2055::DEBUG::2012-08-30 13:10:36,280::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2055::DEBUG::2012-08-30 13:10:36,281::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2055::DEBUG::2012-08-30 13:10:36,281::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2055::DEBUG::2012-08-30 13:10:36,281::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2055::DEBUG::2012-08-30 13:10:36,281::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2055::DEBUG::2012-08-30 13:10:36,282::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2055::DEBUG::2012-08-30 13:10:36,282::task::978::TaskManager.Task::(_decref) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:10:38,564::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:38,565::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2056::DEBUG::2012-08-30 13:10:39,758::task::588::TaskManager.Task::(_updateState) Task=`5ee3b4e7-8e74-418c-9ad5-e3fd332418b1`::moving from state init -> state preparing >Thread-2056::INFO::2012-08-30 13:10:39,759::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2056::INFO::2012-08-30 13:10:39,759::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00157690048218', 'lastCheck': 1346303430.9232769, 'code': 0, 'valid': True}} >Thread-2056::DEBUG::2012-08-30 13:10:39,759::task::1172::TaskManager.Task::(prepare) Task=`5ee3b4e7-8e74-418c-9ad5-e3fd332418b1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00157690048218', 'lastCheck': 1346303430.9232769, 'code': 0, 'valid': True}} >Thread-2056::DEBUG::2012-08-30 13:10:39,759::task::588::TaskManager.Task::(_updateState) Task=`5ee3b4e7-8e74-418c-9ad5-e3fd332418b1`::moving from state preparing -> state finished >Thread-2056::DEBUG::2012-08-30 13:10:39,760::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2056::DEBUG::2012-08-30 13:10:39,760::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2056::DEBUG::2012-08-30 13:10:39,760::task::978::TaskManager.Task::(_decref) Task=`5ee3b4e7-8e74-418c-9ad5-e3fd332418b1`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:10:40,703::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2057::DEBUG::2012-08-30 13:10:43,403::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2057::DEBUG::2012-08-30 13:10:43,403::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:10:43,566::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:45,704::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:45,704::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:48,567::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2059::DEBUG::2012-08-30 13:10:49,508::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2059::DEBUG::2012-08-30 13:10:49,508::task::588::TaskManager.Task::(_updateState) Task=`fcefc378-6b99-4038-845d-eb025e11adfa`::moving from state init -> state preparing >Thread-2059::INFO::2012-08-30 13:10:49,509::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2059::INFO::2012-08-30 13:10:49,509::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2059::DEBUG::2012-08-30 13:10:49,509::task::1172::TaskManager.Task::(prepare) Task=`fcefc378-6b99-4038-845d-eb025e11adfa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2059::DEBUG::2012-08-30 13:10:49,509::task::588::TaskManager.Task::(_updateState) Task=`fcefc378-6b99-4038-845d-eb025e11adfa`::moving from state preparing -> state finished >Thread-2059::DEBUG::2012-08-30 13:10:49,510::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2059::DEBUG::2012-08-30 13:10:49,510::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2059::DEBUG::2012-08-30 13:10:49,510::task::978::TaskManager.Task::(_decref) Task=`fcefc378-6b99-4038-845d-eb025e11adfa`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:10:50,705::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2060::DEBUG::2012-08-30 13:10:53,512::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2060::DEBUG::2012-08-30 13:10:53,513::task::588::TaskManager.Task::(_updateState) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::moving from state init -> state preparing >Thread-2060::INFO::2012-08-30 13:10:53,513::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2060::DEBUG::2012-08-30 13:10:53,513::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0ee072a9-3928-48be-9809-1d1774f8c7c7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2060::DEBUG::2012-08-30 13:10:53,513::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2060::DEBUG::2012-08-30 13:10:53,514::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2060::DEBUG::2012-08-30 13:10:53,514::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0ee072a9-3928-48be-9809-1d1774f8c7c7`::Granted request >Thread-2060::DEBUG::2012-08-30 13:10:53,515::task::817::TaskManager.Task::(resourceAcquired) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2060::DEBUG::2012-08-30 13:10:53,515::task::978::TaskManager.Task::(_decref) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::ref 1 aborting False >Thread-2060::INFO::2012-08-30 13:10:53,518::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2060::DEBUG::2012-08-30 13:10:53,518::task::1172::TaskManager.Task::(prepare) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2060::DEBUG::2012-08-30 13:10:53,519::task::588::TaskManager.Task::(_updateState) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::moving from state preparing -> state finished >Thread-2060::DEBUG::2012-08-30 13:10:53,519::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2060::DEBUG::2012-08-30 13:10:53,519::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2060::DEBUG::2012-08-30 13:10:53,519::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2060::DEBUG::2012-08-30 13:10:53,520::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2060::DEBUG::2012-08-30 13:10:53,520::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2060::DEBUG::2012-08-30 13:10:53,520::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2060::DEBUG::2012-08-30 13:10:53,520::task::978::TaskManager.Task::(_decref) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 13:10:53,568::task::588::TaskManager.Task::(_updateState) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:10:53,568::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:10:53,569::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`06cfd44a-2137-457b-9a60-75b6936bdf83`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:10:53,569::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:10:53,569::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:10:53,569::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`06cfd44a-2137-457b-9a60-75b6936bdf83`::Granted request >Thread-40::DEBUG::2012-08-30 13:10:53,570::task::817::TaskManager.Task::(resourceAcquired) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:10:53,570::task::978::TaskManager.Task::(_decref) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:10:53,571::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:10:53,574::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:10:53,575::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:10:53,575::task::1172::TaskManager.Task::(prepare) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:10:53,575::task::588::TaskManager.Task::(_updateState) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:10:53,575::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:10:53,576::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:10:53,576::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:10:53,576::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:10:53,576::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:10:53,577::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:10:53,577::task::978::TaskManager.Task::(_decref) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:10:53,577::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:53,578::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:53,578::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:10:53,578::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:10:55,706::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:10:56,080::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:10:58,580::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:00,707::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:00,708::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:11:03,087::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:11:03,581::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:05,709::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2064::DEBUG::2012-08-30 13:11:06,694::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2064::DEBUG::2012-08-30 13:11:06,694::task::588::TaskManager.Task::(_updateState) Task=`67ce3f01-e38a-4df2-adf9-fcc740ee133e`::moving from state init -> state preparing >Thread-2064::INFO::2012-08-30 13:11:06,694::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2064::INFO::2012-08-30 13:11:06,695::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2064::DEBUG::2012-08-30 13:11:06,695::task::1172::TaskManager.Task::(prepare) Task=`67ce3f01-e38a-4df2-adf9-fcc740ee133e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2064::DEBUG::2012-08-30 13:11:06,695::task::588::TaskManager.Task::(_updateState) Task=`67ce3f01-e38a-4df2-adf9-fcc740ee133e`::moving from state preparing -> state finished >Thread-2064::DEBUG::2012-08-30 13:11:06,695::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2064::DEBUG::2012-08-30 13:11:06,696::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2064::DEBUG::2012-08-30 13:11:06,696::task::978::TaskManager.Task::(_decref) Task=`67ce3f01-e38a-4df2-adf9-fcc740ee133e`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:11:08,582::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:08,583::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2065::DEBUG::2012-08-30 13:11:09,087::task::588::TaskManager.Task::(_updateState) Task=`becf2e26-3bee-4db5-854b-836e5169c3ec`::moving from state init -> state preparing >Thread-2065::INFO::2012-08-30 13:11:09,088::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2065::INFO::2012-08-30 13:11:09,088::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177311897278', 'lastCheck': 1346303460.9437189, 'code': 0, 'valid': True}} >Thread-2065::DEBUG::2012-08-30 13:11:09,088::task::1172::TaskManager.Task::(prepare) Task=`becf2e26-3bee-4db5-854b-836e5169c3ec`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177311897278', 'lastCheck': 1346303460.9437189, 'code': 0, 'valid': True}} >Thread-2065::DEBUG::2012-08-30 13:11:09,088::task::588::TaskManager.Task::(_updateState) Task=`becf2e26-3bee-4db5-854b-836e5169c3ec`::moving from state preparing -> state finished >Thread-2065::DEBUG::2012-08-30 13:11:09,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2065::DEBUG::2012-08-30 13:11:09,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2065::DEBUG::2012-08-30 13:11:09,089::task::978::TaskManager.Task::(_decref) Task=`becf2e26-3bee-4db5-854b-836e5169c3ec`::ref 0 aborting False >Thread-2066::DEBUG::2012-08-30 13:11:09,845::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2066::DEBUG::2012-08-30 13:11:09,846::task::588::TaskManager.Task::(_updateState) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::moving from state init -> state preparing >Thread-2066::INFO::2012-08-30 13:11:09,846::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2066::DEBUG::2012-08-30 13:11:09,846::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3d56dc96-20bd-4a42-8802-3743dbb3411a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2066::DEBUG::2012-08-30 13:11:09,846::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2066::DEBUG::2012-08-30 13:11:09,847::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2066::DEBUG::2012-08-30 13:11:09,847::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3d56dc96-20bd-4a42-8802-3743dbb3411a`::Granted request >Thread-2066::DEBUG::2012-08-30 13:11:09,847::task::817::TaskManager.Task::(resourceAcquired) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2066::DEBUG::2012-08-30 13:11:09,847::task::978::TaskManager.Task::(_decref) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::ref 1 aborting False >Thread-2066::INFO::2012-08-30 13:11:09,850::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2066::DEBUG::2012-08-30 13:11:09,851::task::1172::TaskManager.Task::(prepare) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2066::DEBUG::2012-08-30 13:11:09,851::task::588::TaskManager.Task::(_updateState) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::moving from state preparing -> state finished >Thread-2066::DEBUG::2012-08-30 13:11:09,851::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2066::DEBUG::2012-08-30 13:11:09,851::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2066::DEBUG::2012-08-30 13:11:09,852::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2066::DEBUG::2012-08-30 13:11:09,852::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2066::DEBUG::2012-08-30 13:11:09,852::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2066::DEBUG::2012-08-30 13:11:09,852::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2066::DEBUG::2012-08-30 13:11:09,853::task::978::TaskManager.Task::(_decref) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:11:10,710::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2067::DEBUG::2012-08-30 13:11:12,245::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2067::DEBUG::2012-08-30 13:11:12,245::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:11:13,584::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:15,711::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:15,711::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:18,585::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:20,712::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:23,587::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:23,587::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2070::DEBUG::2012-08-30 13:11:23,854::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2070::DEBUG::2012-08-30 13:11:23,854::task::588::TaskManager.Task::(_updateState) Task=`bb714a8b-cb94-4f55-90d8-3ce27664fded`::moving from state init -> state preparing >Thread-2070::INFO::2012-08-30 13:11:23,855::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2070::INFO::2012-08-30 13:11:23,855::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2070::DEBUG::2012-08-30 13:11:23,855::task::1172::TaskManager.Task::(prepare) Task=`bb714a8b-cb94-4f55-90d8-3ce27664fded`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2070::DEBUG::2012-08-30 13:11:23,856::task::588::TaskManager.Task::(_updateState) Task=`bb714a8b-cb94-4f55-90d8-3ce27664fded`::moving from state preparing -> state finished >Thread-2070::DEBUG::2012-08-30 13:11:23,856::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2070::DEBUG::2012-08-30 13:11:23,856::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2070::DEBUG::2012-08-30 13:11:23,856::task::978::TaskManager.Task::(_decref) Task=`bb714a8b-cb94-4f55-90d8-3ce27664fded`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:11:25,713::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:11:26,109::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2071::DEBUG::2012-08-30 13:11:27,532::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2071::DEBUG::2012-08-30 13:11:27,533::task::588::TaskManager.Task::(_updateState) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::moving from state init -> state preparing >Thread-2071::INFO::2012-08-30 13:11:27,533::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2071::DEBUG::2012-08-30 13:11:27,533::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`da83211b-bc78-4b25-a88d-4b60870bfcba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2071::DEBUG::2012-08-30 13:11:27,533::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2071::DEBUG::2012-08-30 13:11:27,534::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2071::DEBUG::2012-08-30 13:11:27,534::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`da83211b-bc78-4b25-a88d-4b60870bfcba`::Granted request >Thread-2071::DEBUG::2012-08-30 13:11:27,534::task::817::TaskManager.Task::(resourceAcquired) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2071::DEBUG::2012-08-30 13:11:27,535::task::978::TaskManager.Task::(_decref) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::ref 1 aborting False >Thread-2071::INFO::2012-08-30 13:11:27,537::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2071::DEBUG::2012-08-30 13:11:27,538::task::1172::TaskManager.Task::(prepare) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2071::DEBUG::2012-08-30 13:11:27,538::task::588::TaskManager.Task::(_updateState) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::moving from state preparing -> state finished >Thread-2071::DEBUG::2012-08-30 13:11:27,538::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2071::DEBUG::2012-08-30 13:11:27,538::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2071::DEBUG::2012-08-30 13:11:27,539::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2071::DEBUG::2012-08-30 13:11:27,539::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2071::DEBUG::2012-08-30 13:11:27,539::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2071::DEBUG::2012-08-30 13:11:27,539::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2071::DEBUG::2012-08-30 13:11:27,540::task::978::TaskManager.Task::(_decref) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:11:28,588::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:11:30,714::task::588::TaskManager.Task::(_updateState) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:11:30,715::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:11:30,715::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d6646948-9c83-4d51-9d95-5c7c30d55581`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:11:30,715::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:11:30,715::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:11:30,716::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d6646948-9c83-4d51-9d95-5c7c30d55581`::Granted request >Thread-104::DEBUG::2012-08-30 13:11:30,716::task::817::TaskManager.Task::(resourceAcquired) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:11:30,716::task::978::TaskManager.Task::(_decref) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:11:30,718::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:11:30,720::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:11:30,721::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:11:30,721::task::1172::TaskManager.Task::(prepare) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:11:30,721::task::588::TaskManager.Task::(_updateState) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:11:30,722::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:11:30,722::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:11:30,722::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:11:30,722::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:11:30,723::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:11:30,723::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:11:30,723::task::978::TaskManager.Task::(_decref) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:11:30,723::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:30,724::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:30,724::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:30,725::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:11:33,116::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:11:33,590::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:35,726::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:38,591::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:38,591::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2074::DEBUG::2012-08-30 13:11:39,160::task::588::TaskManager.Task::(_updateState) Task=`869c3305-a46d-466f-8b67-e2a2811f6035`::moving from state init -> state preparing >Thread-2074::INFO::2012-08-30 13:11:39,160::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2074::INFO::2012-08-30 13:11:39,161::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346303490.9625361, 'code': 0, 'valid': True}} >Thread-2074::DEBUG::2012-08-30 13:11:39,161::task::1172::TaskManager.Task::(prepare) Task=`869c3305-a46d-466f-8b67-e2a2811f6035`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346303490.9625361, 'code': 0, 'valid': True}} >Thread-2074::DEBUG::2012-08-30 13:11:39,161::task::588::TaskManager.Task::(_updateState) Task=`869c3305-a46d-466f-8b67-e2a2811f6035`::moving from state preparing -> state finished >Thread-2074::DEBUG::2012-08-30 13:11:39,161::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2074::DEBUG::2012-08-30 13:11:39,162::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2074::DEBUG::2012-08-30 13:11:39,162::task::978::TaskManager.Task::(_decref) Task=`869c3305-a46d-466f-8b67-e2a2811f6035`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:11:40,727::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2075::DEBUG::2012-08-30 13:11:40,754::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2075::DEBUG::2012-08-30 13:11:40,755::task::588::TaskManager.Task::(_updateState) Task=`a73088d1-2285-4be3-b578-909e85ba9749`::moving from state init -> state preparing >Thread-2075::INFO::2012-08-30 13:11:40,755::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2075::INFO::2012-08-30 13:11:40,755::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2075::DEBUG::2012-08-30 13:11:40,756::task::1172::TaskManager.Task::(prepare) Task=`a73088d1-2285-4be3-b578-909e85ba9749`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2075::DEBUG::2012-08-30 13:11:40,756::task::588::TaskManager.Task::(_updateState) Task=`a73088d1-2285-4be3-b578-909e85ba9749`::moving from state preparing -> state finished >Thread-2075::DEBUG::2012-08-30 13:11:40,756::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2075::DEBUG::2012-08-30 13:11:40,756::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2075::DEBUG::2012-08-30 13:11:40,756::task::978::TaskManager.Task::(_decref) Task=`a73088d1-2285-4be3-b578-909e85ba9749`::ref 0 aborting False >Thread-2076::DEBUG::2012-08-30 13:11:42,334::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2076::DEBUG::2012-08-30 13:11:42,334::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:11:43,592::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2077::DEBUG::2012-08-30 13:11:43,999::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2077::DEBUG::2012-08-30 13:11:44,000::task::588::TaskManager.Task::(_updateState) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::moving from state init -> state preparing >Thread-2077::INFO::2012-08-30 13:11:44,000::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2077::DEBUG::2012-08-30 13:11:44,000::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f288e760-4acb-4d2c-984a-875252619b48`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2077::DEBUG::2012-08-30 13:11:44,001::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2077::DEBUG::2012-08-30 13:11:44,001::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2077::DEBUG::2012-08-30 13:11:44,001::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f288e760-4acb-4d2c-984a-875252619b48`::Granted request >Thread-2077::DEBUG::2012-08-30 13:11:44,001::task::817::TaskManager.Task::(resourceAcquired) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2077::DEBUG::2012-08-30 13:11:44,002::task::978::TaskManager.Task::(_decref) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::ref 1 aborting False >Thread-2077::INFO::2012-08-30 13:11:44,005::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2077::DEBUG::2012-08-30 13:11:44,005::task::1172::TaskManager.Task::(prepare) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2077::DEBUG::2012-08-30 13:11:44,005::task::588::TaskManager.Task::(_updateState) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::moving from state preparing -> state finished >Thread-2077::DEBUG::2012-08-30 13:11:44,005::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2077::DEBUG::2012-08-30 13:11:44,006::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2077::DEBUG::2012-08-30 13:11:44,006::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2077::DEBUG::2012-08-30 13:11:44,006::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2077::DEBUG::2012-08-30 13:11:44,006::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2077::DEBUG::2012-08-30 13:11:44,007::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2077::DEBUG::2012-08-30 13:11:44,007::task::978::TaskManager.Task::(_decref) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:11:45,728::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:45,728::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:48,594::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:50,729::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:11:53,595::task::588::TaskManager.Task::(_updateState) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:11:53,595::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:11:53,596::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`10a218d6-7b5c-4415-94af-b87d9262e838`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:11:53,596::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:11:53,596::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:11:53,597::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`10a218d6-7b5c-4415-94af-b87d9262e838`::Granted request >Thread-40::DEBUG::2012-08-30 13:11:53,597::task::817::TaskManager.Task::(resourceAcquired) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:11:53,597::task::978::TaskManager.Task::(_decref) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:11:53,599::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:11:53,601::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:11:53,602::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:11:53,602::task::1172::TaskManager.Task::(prepare) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:11:53,603::task::588::TaskManager.Task::(_updateState) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:11:53,603::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:11:53,603::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:11:53,603::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:11:53,604::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:11:53,604::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:11:53,604::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:11:53,604::task::978::TaskManager.Task::(_decref) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:11:53,605::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:53,605::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:53,606::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:11:53,606::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:11:55,730::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:11:56,134::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2080::DEBUG::2012-08-30 13:11:57,173::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2080::DEBUG::2012-08-30 13:11:57,173::task::588::TaskManager.Task::(_updateState) Task=`544ed687-8158-4eec-82c4-22ff4d454049`::moving from state init -> state preparing >Thread-2080::INFO::2012-08-30 13:11:57,173::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2080::INFO::2012-08-30 13:11:57,174::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2080::DEBUG::2012-08-30 13:11:57,174::task::1172::TaskManager.Task::(prepare) Task=`544ed687-8158-4eec-82c4-22ff4d454049`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2080::DEBUG::2012-08-30 13:11:57,174::task::588::TaskManager.Task::(_updateState) Task=`544ed687-8158-4eec-82c4-22ff4d454049`::moving from state preparing -> state finished >Thread-2080::DEBUG::2012-08-30 13:11:57,174::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2080::DEBUG::2012-08-30 13:11:57,175::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2080::DEBUG::2012-08-30 13:11:57,175::task::978::TaskManager.Task::(_decref) Task=`544ed687-8158-4eec-82c4-22ff4d454049`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:11:58,607::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2082::DEBUG::2012-08-30 13:12:00,335::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2082::DEBUG::2012-08-30 13:12:00,335::task::588::TaskManager.Task::(_updateState) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::moving from state init -> state preparing >Thread-2082::INFO::2012-08-30 13:12:00,335::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2082::DEBUG::2012-08-30 13:12:00,336::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d8069c73-14d8-4421-9f23-a5b9944f0f32`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2082::DEBUG::2012-08-30 13:12:00,336::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2082::DEBUG::2012-08-30 13:12:00,336::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2082::DEBUG::2012-08-30 13:12:00,336::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d8069c73-14d8-4421-9f23-a5b9944f0f32`::Granted request >Thread-2082::DEBUG::2012-08-30 13:12:00,337::task::817::TaskManager.Task::(resourceAcquired) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2082::DEBUG::2012-08-30 13:12:00,337::task::978::TaskManager.Task::(_decref) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::ref 1 aborting False >Thread-2082::INFO::2012-08-30 13:12:00,340::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2082::DEBUG::2012-08-30 13:12:00,340::task::1172::TaskManager.Task::(prepare) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2082::DEBUG::2012-08-30 13:12:00,341::task::588::TaskManager.Task::(_updateState) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::moving from state preparing -> state finished >Thread-2082::DEBUG::2012-08-30 13:12:00,341::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2082::DEBUG::2012-08-30 13:12:00,341::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2082::DEBUG::2012-08-30 13:12:00,341::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2082::DEBUG::2012-08-30 13:12:00,342::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2082::DEBUG::2012-08-30 13:12:00,342::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2082::DEBUG::2012-08-30 13:12:00,342::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2082::DEBUG::2012-08-30 13:12:00,342::task::978::TaskManager.Task::(_decref) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:12:00,731::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:00,732::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:12:03,140::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:12:03,608::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:05,732::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2084::DEBUG::2012-08-30 13:12:08,257::task::588::TaskManager.Task::(_updateState) Task=`661dff62-133b-4ab4-b21b-2a984aa974c8`::moving from state init -> state preparing >Thread-2084::INFO::2012-08-30 13:12:08,257::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2084::INFO::2012-08-30 13:12:08,257::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177812576294', 'lastCheck': 1346303520.9828539, 'code': 0, 'valid': True}} >Thread-2084::DEBUG::2012-08-30 13:12:08,258::task::1172::TaskManager.Task::(prepare) Task=`661dff62-133b-4ab4-b21b-2a984aa974c8`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177812576294', 'lastCheck': 1346303520.9828539, 'code': 0, 'valid': True}} >Thread-2084::DEBUG::2012-08-30 13:12:08,258::task::588::TaskManager.Task::(_updateState) Task=`661dff62-133b-4ab4-b21b-2a984aa974c8`::moving from state preparing -> state finished >Thread-2084::DEBUG::2012-08-30 13:12:08,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2084::DEBUG::2012-08-30 13:12:08,258::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2084::DEBUG::2012-08-30 13:12:08,259::task::978::TaskManager.Task::(_decref) Task=`661dff62-133b-4ab4-b21b-2a984aa974c8`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:12:08,609::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:08,610::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:10,733::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2085::DEBUG::2012-08-30 13:12:11,737::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2085::DEBUG::2012-08-30 13:12:11,738::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-2086::DEBUG::2012-08-30 13:12:13,307::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2086::DEBUG::2012-08-30 13:12:13,307::task::588::TaskManager.Task::(_updateState) Task=`9b2ede0b-1ea5-4bd8-ab14-4de7be082dc7`::moving from state init -> state preparing >Thread-2086::INFO::2012-08-30 13:12:13,307::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2086::INFO::2012-08-30 13:12:13,308::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2086::DEBUG::2012-08-30 13:12:13,308::task::1172::TaskManager.Task::(prepare) Task=`9b2ede0b-1ea5-4bd8-ab14-4de7be082dc7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2086::DEBUG::2012-08-30 13:12:13,308::task::588::TaskManager.Task::(_updateState) Task=`9b2ede0b-1ea5-4bd8-ab14-4de7be082dc7`::moving from state preparing -> state finished >Thread-2086::DEBUG::2012-08-30 13:12:13,308::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2086::DEBUG::2012-08-30 13:12:13,309::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2086::DEBUG::2012-08-30 13:12:13,309::task::978::TaskManager.Task::(_decref) Task=`9b2ede0b-1ea5-4bd8-ab14-4de7be082dc7`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:12:13,611::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:15,734::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:15,735::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2087::DEBUG::2012-08-30 13:12:16,457::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2087::DEBUG::2012-08-30 13:12:16,458::task::588::TaskManager.Task::(_updateState) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::moving from state init -> state preparing >Thread-2087::INFO::2012-08-30 13:12:16,458::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2087::DEBUG::2012-08-30 13:12:16,458::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f7f6a7bf-3b8f-472a-b521-4424ae549db3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2087::DEBUG::2012-08-30 13:12:16,459::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2087::DEBUG::2012-08-30 13:12:16,459::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2087::DEBUG::2012-08-30 13:12:16,459::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f7f6a7bf-3b8f-472a-b521-4424ae549db3`::Granted request >Thread-2087::DEBUG::2012-08-30 13:12:16,460::task::817::TaskManager.Task::(resourceAcquired) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2087::DEBUG::2012-08-30 13:12:16,460::task::978::TaskManager.Task::(_decref) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::ref 1 aborting False >Thread-2087::INFO::2012-08-30 13:12:16,463::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2087::DEBUG::2012-08-30 13:12:16,463::task::1172::TaskManager.Task::(prepare) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2087::DEBUG::2012-08-30 13:12:16,463::task::588::TaskManager.Task::(_updateState) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::moving from state preparing -> state finished >Thread-2087::DEBUG::2012-08-30 13:12:16,463::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2087::DEBUG::2012-08-30 13:12:16,464::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2087::DEBUG::2012-08-30 13:12:16,464::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2087::DEBUG::2012-08-30 13:12:16,464::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2087::DEBUG::2012-08-30 13:12:16,464::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2087::DEBUG::2012-08-30 13:12:16,465::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2087::DEBUG::2012-08-30 13:12:16,465::task::978::TaskManager.Task::(_decref) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:12:18,613::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:20,736::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:23,614::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:23,614::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:25,736::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:12:26,154::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:12:28,616::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2091::DEBUG::2012-08-30 13:12:29,638::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2091::DEBUG::2012-08-30 13:12:29,639::task::588::TaskManager.Task::(_updateState) Task=`5be5b960-b878-4c4d-82b5-47dbae1152de`::moving from state init -> state preparing >Thread-2091::INFO::2012-08-30 13:12:29,639::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2091::INFO::2012-08-30 13:12:29,639::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2091::DEBUG::2012-08-30 13:12:29,639::task::1172::TaskManager.Task::(prepare) Task=`5be5b960-b878-4c4d-82b5-47dbae1152de`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2091::DEBUG::2012-08-30 13:12:29,640::task::588::TaskManager.Task::(_updateState) Task=`5be5b960-b878-4c4d-82b5-47dbae1152de`::moving from state preparing -> state finished >Thread-2091::DEBUG::2012-08-30 13:12:29,640::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2091::DEBUG::2012-08-30 13:12:29,640::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2091::DEBUG::2012-08-30 13:12:29,640::task::978::TaskManager.Task::(_decref) Task=`5be5b960-b878-4c4d-82b5-47dbae1152de`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 13:12:30,737::task::588::TaskManager.Task::(_updateState) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:12:30,738::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:12:30,738::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bb2c0319-ef19-4827-a1ec-2ff661e12bc1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:12:30,738::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:12:30,739::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:12:30,739::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bb2c0319-ef19-4827-a1ec-2ff661e12bc1`::Granted request >Thread-104::DEBUG::2012-08-30 13:12:30,739::task::817::TaskManager.Task::(resourceAcquired) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:12:30,739::task::978::TaskManager.Task::(_decref) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:12:30,741::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:12:30,743::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:12:30,744::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:12:30,744::task::1172::TaskManager.Task::(prepare) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:12:30,745::task::588::TaskManager.Task::(_updateState) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:12:30,745::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:12:30,745::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:12:30,745::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:12:30,746::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:12:30,746::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:12:30,746::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:12:30,746::task::978::TaskManager.Task::(_decref) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:12:30,747::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:30,747::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:30,748::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:30,748::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2093::DEBUG::2012-08-30 13:12:32,900::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2093::DEBUG::2012-08-30 13:12:32,900::task::588::TaskManager.Task::(_updateState) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::moving from state init -> state preparing >Thread-2093::INFO::2012-08-30 13:12:32,901::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2093::DEBUG::2012-08-30 13:12:32,901::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c2d85981-8ba7-4315-8e1b-f70731fa336b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2093::DEBUG::2012-08-30 13:12:32,901::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2093::DEBUG::2012-08-30 13:12:32,901::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2093::DEBUG::2012-08-30 13:12:32,902::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c2d85981-8ba7-4315-8e1b-f70731fa336b`::Granted request >Thread-2093::DEBUG::2012-08-30 13:12:32,902::task::817::TaskManager.Task::(resourceAcquired) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2093::DEBUG::2012-08-30 13:12:32,902::task::978::TaskManager.Task::(_decref) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::ref 1 aborting False >Thread-2093::INFO::2012-08-30 13:12:32,905::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2093::DEBUG::2012-08-30 13:12:32,905::task::1172::TaskManager.Task::(prepare) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2093::DEBUG::2012-08-30 13:12:32,906::task::588::TaskManager.Task::(_updateState) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::moving from state preparing -> state finished >Thread-2093::DEBUG::2012-08-30 13:12:32,906::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2093::DEBUG::2012-08-30 13:12:32,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2093::DEBUG::2012-08-30 13:12:32,906::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2093::DEBUG::2012-08-30 13:12:32,907::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2093::DEBUG::2012-08-30 13:12:32,907::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2093::DEBUG::2012-08-30 13:12:32,907::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2093::DEBUG::2012-08-30 13:12:32,907::task::978::TaskManager.Task::(_decref) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:12:33,160::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:12:33,617::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:35,749::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2094::DEBUG::2012-08-30 13:12:37,079::task::588::TaskManager.Task::(_updateState) Task=`87dfcd3a-1596-4ffb-9601-6cd89dab6a9c`::moving from state init -> state preparing >Thread-2094::INFO::2012-08-30 13:12:37,079::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2094::INFO::2012-08-30 13:12:37,080::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162577629089', 'lastCheck': 1346303551.001581, 'code': 0, 'valid': True}} >Thread-2094::DEBUG::2012-08-30 13:12:37,080::task::1172::TaskManager.Task::(prepare) Task=`87dfcd3a-1596-4ffb-9601-6cd89dab6a9c`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162577629089', 'lastCheck': 1346303551.001581, 'code': 0, 'valid': True}} >Thread-2094::DEBUG::2012-08-30 13:12:37,080::task::588::TaskManager.Task::(_updateState) Task=`87dfcd3a-1596-4ffb-9601-6cd89dab6a9c`::moving from state preparing -> state finished >Thread-2094::DEBUG::2012-08-30 13:12:37,080::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2094::DEBUG::2012-08-30 13:12:37,081::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2094::DEBUG::2012-08-30 13:12:37,081::task::978::TaskManager.Task::(_decref) Task=`87dfcd3a-1596-4ffb-9601-6cd89dab6a9c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:12:38,618::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:38,619::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2095::DEBUG::2012-08-30 13:12:40,231::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2095::DEBUG::2012-08-30 13:12:40,231::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:12:40,750::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:43,620::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2097::DEBUG::2012-08-30 13:12:45,462::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2097::DEBUG::2012-08-30 13:12:45,462::task::588::TaskManager.Task::(_updateState) Task=`58bb7e7d-14c2-46e6-bca3-f1c142aa1db1`::moving from state init -> state preparing >Thread-2097::INFO::2012-08-30 13:12:45,462::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2097::INFO::2012-08-30 13:12:45,463::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2097::DEBUG::2012-08-30 13:12:45,463::task::1172::TaskManager.Task::(prepare) Task=`58bb7e7d-14c2-46e6-bca3-f1c142aa1db1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2097::DEBUG::2012-08-30 13:12:45,463::task::588::TaskManager.Task::(_updateState) Task=`58bb7e7d-14c2-46e6-bca3-f1c142aa1db1`::moving from state preparing -> state finished >Thread-2097::DEBUG::2012-08-30 13:12:45,463::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2097::DEBUG::2012-08-30 13:12:45,464::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2097::DEBUG::2012-08-30 13:12:45,464::task::978::TaskManager.Task::(_decref) Task=`58bb7e7d-14c2-46e6-bca3-f1c142aa1db1`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:12:45,751::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:45,751::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:48,621::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2098::DEBUG::2012-08-30 13:12:48,803::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2098::DEBUG::2012-08-30 13:12:48,804::task::588::TaskManager.Task::(_updateState) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::moving from state init -> state preparing >Thread-2098::INFO::2012-08-30 13:12:48,804::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2098::DEBUG::2012-08-30 13:12:48,804::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`77545583-7d2f-42e7-8ee9-c6295ab1055a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2098::DEBUG::2012-08-30 13:12:48,804::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2098::DEBUG::2012-08-30 13:12:48,805::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2098::DEBUG::2012-08-30 13:12:48,805::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`77545583-7d2f-42e7-8ee9-c6295ab1055a`::Granted request >Thread-2098::DEBUG::2012-08-30 13:12:48,805::task::817::TaskManager.Task::(resourceAcquired) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2098::DEBUG::2012-08-30 13:12:48,805::task::978::TaskManager.Task::(_decref) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::ref 1 aborting False >Thread-2098::INFO::2012-08-30 13:12:48,808::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2098::DEBUG::2012-08-30 13:12:48,809::task::1172::TaskManager.Task::(prepare) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2098::DEBUG::2012-08-30 13:12:48,809::task::588::TaskManager.Task::(_updateState) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::moving from state preparing -> state finished >Thread-2098::DEBUG::2012-08-30 13:12:48,809::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2098::DEBUG::2012-08-30 13:12:48,809::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2098::DEBUG::2012-08-30 13:12:48,810::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2098::DEBUG::2012-08-30 13:12:48,810::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2098::DEBUG::2012-08-30 13:12:48,810::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2098::DEBUG::2012-08-30 13:12:48,810::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2098::DEBUG::2012-08-30 13:12:48,811::task::978::TaskManager.Task::(_decref) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:12:50,752::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:12:53,622::task::588::TaskManager.Task::(_updateState) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:12:53,622::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:12:53,623::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`71070b69-0bad-4e0c-98f5-2857f246babe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:12:53,623::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:12:53,624::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:12:53,624::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`71070b69-0bad-4e0c-98f5-2857f246babe`::Granted request >Thread-40::DEBUG::2012-08-30 13:12:53,624::task::817::TaskManager.Task::(resourceAcquired) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:12:53,624::task::978::TaskManager.Task::(_decref) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:12:53,626::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:12:53,628::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:12:53,629::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:12:53,630::task::1172::TaskManager.Task::(prepare) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:12:53,630::task::588::TaskManager.Task::(_updateState) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:12:53,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:12:53,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:12:53,631::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:12:53,631::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:12:53,631::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:12:53,631::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:12:53,631::task::978::TaskManager.Task::(_decref) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:12:53,632::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:53,632::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:53,633::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:12:53,633::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:12:55,753::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:12:56,179::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:12:58,634::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:00,754::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:00,754::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2102::DEBUG::2012-08-30 13:13:02,019::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2102::DEBUG::2012-08-30 13:13:02,019::task::588::TaskManager.Task::(_updateState) Task=`8d37e533-12c7-495c-b064-ac5d01837169`::moving from state init -> state preparing >Thread-2102::INFO::2012-08-30 13:13:02,019::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2102::INFO::2012-08-30 13:13:02,020::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2102::DEBUG::2012-08-30 13:13:02,020::task::1172::TaskManager.Task::(prepare) Task=`8d37e533-12c7-495c-b064-ac5d01837169`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2102::DEBUG::2012-08-30 13:13:02,020::task::588::TaskManager.Task::(_updateState) Task=`8d37e533-12c7-495c-b064-ac5d01837169`::moving from state preparing -> state finished >Thread-2102::DEBUG::2012-08-30 13:13:02,020::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2102::DEBUG::2012-08-30 13:13:02,021::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2102::DEBUG::2012-08-30 13:13:02,021::task::978::TaskManager.Task::(_decref) Task=`8d37e533-12c7-495c-b064-ac5d01837169`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:13:03,185::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:13:03,636::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2103::DEBUG::2012-08-30 13:13:05,178::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2104::DEBUG::2012-08-30 13:13:05,179::task::588::TaskManager.Task::(_updateState) Task=`d827489b-d98c-4e9e-8b85-32babf3a2f56`::moving from state init -> state preparing >Thread-2103::DEBUG::2012-08-30 13:13:05,179::task::588::TaskManager.Task::(_updateState) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::moving from state init -> state preparing >Thread-2104::INFO::2012-08-30 13:13:05,179::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2103::INFO::2012-08-30 13:13:05,179::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2104::INFO::2012-08-30 13:13:05,180::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177693367004', 'lastCheck': 1346303581.0220251, 'code': 0, 'valid': True}} >Thread-2103::DEBUG::2012-08-30 13:13:05,180::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0c4c7978-f91d-4c86-ac29-9ef6f1de79d2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2104::DEBUG::2012-08-30 13:13:05,180::task::1172::TaskManager.Task::(prepare) Task=`d827489b-d98c-4e9e-8b85-32babf3a2f56`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177693367004', 'lastCheck': 1346303581.0220251, 'code': 0, 'valid': True}} >Thread-2103::DEBUG::2012-08-30 13:13:05,180::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2104::DEBUG::2012-08-30 13:13:05,181::task::588::TaskManager.Task::(_updateState) Task=`d827489b-d98c-4e9e-8b85-32babf3a2f56`::moving from state preparing -> state finished >Thread-2103::DEBUG::2012-08-30 13:13:05,181::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2104::DEBUG::2012-08-30 13:13:05,181::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2103::DEBUG::2012-08-30 13:13:05,181::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0c4c7978-f91d-4c86-ac29-9ef6f1de79d2`::Granted request >Thread-2104::DEBUG::2012-08-30 13:13:05,181::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2103::DEBUG::2012-08-30 13:13:05,182::task::817::TaskManager.Task::(resourceAcquired) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2104::DEBUG::2012-08-30 13:13:05,182::task::978::TaskManager.Task::(_decref) Task=`d827489b-d98c-4e9e-8b85-32babf3a2f56`::ref 0 aborting False >Thread-2103::DEBUG::2012-08-30 13:13:05,182::task::978::TaskManager.Task::(_decref) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::ref 1 aborting False >Thread-2103::INFO::2012-08-30 13:13:05,187::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2103::DEBUG::2012-08-30 13:13:05,187::task::1172::TaskManager.Task::(prepare) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2103::DEBUG::2012-08-30 13:13:05,187::task::588::TaskManager.Task::(_updateState) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::moving from state preparing -> state finished >Thread-2103::DEBUG::2012-08-30 13:13:05,188::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2103::DEBUG::2012-08-30 13:13:05,188::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2103::DEBUG::2012-08-30 13:13:05,188::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2103::DEBUG::2012-08-30 13:13:05,188::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2103::DEBUG::2012-08-30 13:13:05,189::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2103::DEBUG::2012-08-30 13:13:05,189::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2103::DEBUG::2012-08-30 13:13:05,189::task::978::TaskManager.Task::(_decref) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:13:05,755::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2105::DEBUG::2012-08-30 13:13:08,363::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2105::DEBUG::2012-08-30 13:13:08,363::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:13:08,637::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:08,637::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:10,756::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:13,638::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:15,757::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:15,758::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:18,640::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2107::DEBUG::2012-08-30 13:13:19,442::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2107::DEBUG::2012-08-30 13:13:19,443::task::588::TaskManager.Task::(_updateState) Task=`d2c8ae02-5d20-4ca7-83af-9c8803dfcaf6`::moving from state init -> state preparing >Thread-2107::INFO::2012-08-30 13:13:19,443::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2107::INFO::2012-08-30 13:13:19,444::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2107::DEBUG::2012-08-30 13:13:19,444::task::1172::TaskManager.Task::(prepare) Task=`d2c8ae02-5d20-4ca7-83af-9c8803dfcaf6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2107::DEBUG::2012-08-30 13:13:19,444::task::588::TaskManager.Task::(_updateState) Task=`d2c8ae02-5d20-4ca7-83af-9c8803dfcaf6`::moving from state preparing -> state finished >Thread-2107::DEBUG::2012-08-30 13:13:19,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2107::DEBUG::2012-08-30 13:13:19,445::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2107::DEBUG::2012-08-30 13:13:19,445::task::978::TaskManager.Task::(_decref) Task=`d2c8ae02-5d20-4ca7-83af-9c8803dfcaf6`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:13:20,758::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:23,641::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:23,642::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2109::DEBUG::2012-08-30 13:13:24,181::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2109::DEBUG::2012-08-30 13:13:24,182::task::588::TaskManager.Task::(_updateState) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::moving from state init -> state preparing >Thread-2109::INFO::2012-08-30 13:13:24,182::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2109::DEBUG::2012-08-30 13:13:24,182::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`de5fdbfd-cea7-4751-ac24-ac975612cfe6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2109::DEBUG::2012-08-30 13:13:24,182::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2109::DEBUG::2012-08-30 13:13:24,183::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2109::DEBUG::2012-08-30 13:13:24,183::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`de5fdbfd-cea7-4751-ac24-ac975612cfe6`::Granted request >Thread-2109::DEBUG::2012-08-30 13:13:24,183::task::817::TaskManager.Task::(resourceAcquired) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2109::DEBUG::2012-08-30 13:13:24,184::task::978::TaskManager.Task::(_decref) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::ref 1 aborting False >Thread-2109::INFO::2012-08-30 13:13:24,186::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2109::DEBUG::2012-08-30 13:13:24,187::task::1172::TaskManager.Task::(prepare) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2109::DEBUG::2012-08-30 13:13:24,187::task::588::TaskManager.Task::(_updateState) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::moving from state preparing -> state finished >Thread-2109::DEBUG::2012-08-30 13:13:24,187::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2109::DEBUG::2012-08-30 13:13:24,187::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2109::DEBUG::2012-08-30 13:13:24,188::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2109::DEBUG::2012-08-30 13:13:24,188::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2109::DEBUG::2012-08-30 13:13:24,188::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2109::DEBUG::2012-08-30 13:13:24,188::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2109::DEBUG::2012-08-30 13:13:24,189::task::978::TaskManager.Task::(_decref) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:13:25,759::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:13:26,204::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:13:28,643::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:13:30,760::task::588::TaskManager.Task::(_updateState) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:13:30,761::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:13:30,761::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`24f2d2cb-a228-49fc-89f6-b9da26b1d806`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:13:30,761::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:13:30,762::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:13:30,762::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`24f2d2cb-a228-49fc-89f6-b9da26b1d806`::Granted request >Thread-104::DEBUG::2012-08-30 13:13:30,762::task::817::TaskManager.Task::(resourceAcquired) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:13:30,762::task::978::TaskManager.Task::(_decref) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:13:30,764::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:13:30,766::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:13:30,767::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:13:30,767::task::1172::TaskManager.Task::(prepare) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:13:30,768::task::588::TaskManager.Task::(_updateState) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:13:30,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:13:30,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:13:30,768::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:13:30,769::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:13:30,769::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:13:30,769::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:13:30,769::task::978::TaskManager.Task::(_decref) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:13:30,770::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:30,770::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:30,771::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:30,771::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:13:33,211::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:13:33,645::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:35,772::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:38,646::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:38,646::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2112::DEBUG::2012-08-30 13:13:39,538::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2112::DEBUG::2012-08-30 13:13:39,539::task::588::TaskManager.Task::(_updateState) Task=`d64154fa-d839-4575-a519-ff5151f6689e`::moving from state init -> state preparing >Thread-2112::INFO::2012-08-30 13:13:39,539::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2112::INFO::2012-08-30 13:13:39,539::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2112::DEBUG::2012-08-30 13:13:39,540::task::1172::TaskManager.Task::(prepare) Task=`d64154fa-d839-4575-a519-ff5151f6689e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2112::DEBUG::2012-08-30 13:13:39,540::task::588::TaskManager.Task::(_updateState) Task=`d64154fa-d839-4575-a519-ff5151f6689e`::moving from state preparing -> state finished >Thread-2112::DEBUG::2012-08-30 13:13:39,540::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2112::DEBUG::2012-08-30 13:13:39,540::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2112::DEBUG::2012-08-30 13:13:39,540::task::978::TaskManager.Task::(_decref) Task=`d64154fa-d839-4575-a519-ff5151f6689e`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:13:40,773::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2113::DEBUG::2012-08-30 13:13:42,397::task::588::TaskManager.Task::(_updateState) Task=`c4c60f6f-754f-4185-a930-85c5d308ca67`::moving from state init -> state preparing >Thread-2113::INFO::2012-08-30 13:13:42,397::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2113::INFO::2012-08-30 13:13:42,397::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016348361969', 'lastCheck': 1346303621.047157, 'code': 0, 'valid': True}} >Thread-2113::DEBUG::2012-08-30 13:13:42,397::task::1172::TaskManager.Task::(prepare) Task=`c4c60f6f-754f-4185-a930-85c5d308ca67`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016348361969', 'lastCheck': 1346303621.047157, 'code': 0, 'valid': True}} >Thread-2113::DEBUG::2012-08-30 13:13:42,398::task::588::TaskManager.Task::(_updateState) Task=`c4c60f6f-754f-4185-a930-85c5d308ca67`::moving from state preparing -> state finished >Thread-2113::DEBUG::2012-08-30 13:13:42,398::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2113::DEBUG::2012-08-30 13:13:42,398::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2113::DEBUG::2012-08-30 13:13:42,398::task::978::TaskManager.Task::(_decref) Task=`c4c60f6f-754f-4185-a930-85c5d308ca67`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:13:43,647::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2114::DEBUG::2012-08-30 13:13:45,344::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2114::DEBUG::2012-08-30 13:13:45,345::task::588::TaskManager.Task::(_updateState) Task=`bfd955b7-024d-4862-b982-69069277160b`::moving from state init -> state preparing >Thread-2114::INFO::2012-08-30 13:13:45,345::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2114::DEBUG::2012-08-30 13:13:45,346::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1e24f4ce-df10-4a7b-9516-8fb374427447`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2114::DEBUG::2012-08-30 13:13:45,346::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2114::DEBUG::2012-08-30 13:13:45,346::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2114::DEBUG::2012-08-30 13:13:45,346::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1e24f4ce-df10-4a7b-9516-8fb374427447`::Granted request >Thread-2114::DEBUG::2012-08-30 13:13:45,347::task::817::TaskManager.Task::(resourceAcquired) Task=`bfd955b7-024d-4862-b982-69069277160b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2114::DEBUG::2012-08-30 13:13:45,347::task::978::TaskManager.Task::(_decref) Task=`bfd955b7-024d-4862-b982-69069277160b`::ref 1 aborting False >Thread-2114::INFO::2012-08-30 13:13:45,350::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2114::DEBUG::2012-08-30 13:13:45,351::task::1172::TaskManager.Task::(prepare) Task=`bfd955b7-024d-4862-b982-69069277160b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2114::DEBUG::2012-08-30 13:13:45,351::task::588::TaskManager.Task::(_updateState) Task=`bfd955b7-024d-4862-b982-69069277160b`::moving from state preparing -> state finished >Thread-2114::DEBUG::2012-08-30 13:13:45,351::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2114::DEBUG::2012-08-30 13:13:45,351::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2114::DEBUG::2012-08-30 13:13:45,352::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2114::DEBUG::2012-08-30 13:13:45,352::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2114::DEBUG::2012-08-30 13:13:45,352::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2114::DEBUG::2012-08-30 13:13:45,352::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2114::DEBUG::2012-08-30 13:13:45,352::task::978::TaskManager.Task::(_decref) Task=`bfd955b7-024d-4862-b982-69069277160b`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:13:45,774::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:45,774::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2115::DEBUG::2012-08-30 13:13:48,048::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2115::DEBUG::2012-08-30 13:13:48,048::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:13:48,649::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:50,775::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:13:53,650::task::588::TaskManager.Task::(_updateState) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:13:53,650::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:13:53,651::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f8c0b1d3-788e-43b8-bb1b-345205f4fd28`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:13:53,651::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:13:53,651::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:13:53,651::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f8c0b1d3-788e-43b8-bb1b-345205f4fd28`::Granted request >Thread-40::DEBUG::2012-08-30 13:13:53,652::task::817::TaskManager.Task::(resourceAcquired) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:13:53,652::task::978::TaskManager.Task::(_decref) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:13:53,654::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:13:53,656::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:13:53,657::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:13:53,657::task::1172::TaskManager.Task::(prepare) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:13:53,657::task::588::TaskManager.Task::(_updateState) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:13:53,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:13:53,658::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:13:53,658::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:13:53,658::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:13:53,659::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:13:53,659::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:13:53,659::task::978::TaskManager.Task::(_decref) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:13:53,659::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:53,660::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:53,660::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:13:53,661::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:13:55,776::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:13:56,231::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:13:58,662::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:00,777::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:00,777::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2117::DEBUG::2012-08-30 13:14:01,492::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2117::DEBUG::2012-08-30 13:14:01,492::task::588::TaskManager.Task::(_updateState) Task=`b38c5db0-2d26-49f2-b427-d564fcac11f3`::moving from state init -> state preparing >Thread-2117::INFO::2012-08-30 13:14:01,492::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2117::INFO::2012-08-30 13:14:01,493::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2117::DEBUG::2012-08-30 13:14:01,493::task::1172::TaskManager.Task::(prepare) Task=`b38c5db0-2d26-49f2-b427-d564fcac11f3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2117::DEBUG::2012-08-30 13:14:01,493::task::588::TaskManager.Task::(_updateState) Task=`b38c5db0-2d26-49f2-b427-d564fcac11f3`::moving from state preparing -> state finished >Thread-2117::DEBUG::2012-08-30 13:14:01,493::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2117::DEBUG::2012-08-30 13:14:01,494::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2117::DEBUG::2012-08-30 13:14:01,494::task::978::TaskManager.Task::(_decref) Task=`b38c5db0-2d26-49f2-b427-d564fcac11f3`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:14:03,238::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:14:03,663::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:05,778::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2119::DEBUG::2012-08-30 13:14:08,208::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2119::DEBUG::2012-08-30 13:14:08,208::task::588::TaskManager.Task::(_updateState) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::moving from state init -> state preparing >Thread-2119::INFO::2012-08-30 13:14:08,208::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2119::DEBUG::2012-08-30 13:14:08,209::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`aac84ae2-41d5-4787-9e2a-6c44ed7e4789`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2119::DEBUG::2012-08-30 13:14:08,209::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2119::DEBUG::2012-08-30 13:14:08,209::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2119::DEBUG::2012-08-30 13:14:08,209::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`aac84ae2-41d5-4787-9e2a-6c44ed7e4789`::Granted request >Thread-2119::DEBUG::2012-08-30 13:14:08,210::task::817::TaskManager.Task::(resourceAcquired) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2119::DEBUG::2012-08-30 13:14:08,210::task::978::TaskManager.Task::(_decref) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::ref 1 aborting False >Thread-2119::INFO::2012-08-30 13:14:08,213::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2119::DEBUG::2012-08-30 13:14:08,213::task::1172::TaskManager.Task::(prepare) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2119::DEBUG::2012-08-30 13:14:08,213::task::588::TaskManager.Task::(_updateState) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::moving from state preparing -> state finished >Thread-2119::DEBUG::2012-08-30 13:14:08,214::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2119::DEBUG::2012-08-30 13:14:08,214::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2119::DEBUG::2012-08-30 13:14:08,214::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2119::DEBUG::2012-08-30 13:14:08,214::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2119::DEBUG::2012-08-30 13:14:08,215::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2119::DEBUG::2012-08-30 13:14:08,215::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2119::DEBUG::2012-08-30 13:14:08,215::task::978::TaskManager.Task::(_decref) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:14:08,664::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:08,665::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:10,779::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:13,666::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:15,780::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:15,781::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:18,667::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:20,782::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2122::DEBUG::2012-08-30 13:14:23,045::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2122::DEBUG::2012-08-30 13:14:23,046::task::588::TaskManager.Task::(_updateState) Task=`480e2b5c-287e-462b-8e05-4c22f964b896`::moving from state init -> state preparing >Thread-2122::INFO::2012-08-30 13:14:23,046::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2122::INFO::2012-08-30 13:14:23,047::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2122::DEBUG::2012-08-30 13:14:23,047::task::1172::TaskManager.Task::(prepare) Task=`480e2b5c-287e-462b-8e05-4c22f964b896`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2122::DEBUG::2012-08-30 13:14:23,047::task::588::TaskManager.Task::(_updateState) Task=`480e2b5c-287e-462b-8e05-4c22f964b896`::moving from state preparing -> state finished >Thread-2122::DEBUG::2012-08-30 13:14:23,047::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2122::DEBUG::2012-08-30 13:14:23,047::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2122::DEBUG::2012-08-30 13:14:23,048::task::978::TaskManager.Task::(_decref) Task=`480e2b5c-287e-462b-8e05-4c22f964b896`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:14:23,669::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:23,669::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2123::DEBUG::2012-08-30 13:14:25,528::task::588::TaskManager.Task::(_updateState) Task=`2c42f780-a9da-4662-a20b-350217ddf2b4`::moving from state init -> state preparing >Thread-2123::INFO::2012-08-30 13:14:25,528::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2123::INFO::2012-08-30 13:14:25,528::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163984298706', 'lastCheck': 1346303661.074084, 'code': 0, 'valid': True}} >Thread-2123::DEBUG::2012-08-30 13:14:25,528::task::1172::TaskManager.Task::(prepare) Task=`2c42f780-a9da-4662-a20b-350217ddf2b4`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163984298706', 'lastCheck': 1346303661.074084, 'code': 0, 'valid': True}} >Thread-2123::DEBUG::2012-08-30 13:14:25,529::task::588::TaskManager.Task::(_updateState) Task=`2c42f780-a9da-4662-a20b-350217ddf2b4`::moving from state preparing -> state finished >Thread-2123::DEBUG::2012-08-30 13:14:25,529::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2123::DEBUG::2012-08-30 13:14:25,529::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2123::DEBUG::2012-08-30 13:14:25,529::task::978::TaskManager.Task::(_decref) Task=`2c42f780-a9da-4662-a20b-350217ddf2b4`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:14:25,783::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:14:26,257::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2124::DEBUG::2012-08-30 13:14:27,131::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2124::DEBUG::2012-08-30 13:14:27,132::task::588::TaskManager.Task::(_updateState) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::moving from state init -> state preparing >Thread-2124::INFO::2012-08-30 13:14:27,132::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2124::DEBUG::2012-08-30 13:14:27,132::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ecc57c27-4fe5-448b-8632-6b389bf765e9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2124::DEBUG::2012-08-30 13:14:27,132::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2124::DEBUG::2012-08-30 13:14:27,133::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2124::DEBUG::2012-08-30 13:14:27,133::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ecc57c27-4fe5-448b-8632-6b389bf765e9`::Granted request >Thread-2124::DEBUG::2012-08-30 13:14:27,133::task::817::TaskManager.Task::(resourceAcquired) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2124::DEBUG::2012-08-30 13:14:27,134::task::978::TaskManager.Task::(_decref) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::ref 1 aborting False >Thread-2124::INFO::2012-08-30 13:14:27,137::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2124::DEBUG::2012-08-30 13:14:27,137::task::1172::TaskManager.Task::(prepare) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2124::DEBUG::2012-08-30 13:14:27,137::task::588::TaskManager.Task::(_updateState) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::moving from state preparing -> state finished >Thread-2124::DEBUG::2012-08-30 13:14:27,137::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2124::DEBUG::2012-08-30 13:14:27,137::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2124::DEBUG::2012-08-30 13:14:27,138::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2124::DEBUG::2012-08-30 13:14:27,138::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2124::DEBUG::2012-08-30 13:14:27,138::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2124::DEBUG::2012-08-30 13:14:27,138::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2124::DEBUG::2012-08-30 13:14:27,139::task::978::TaskManager.Task::(_decref) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:14:28,671::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2125::DEBUG::2012-08-30 13:14:28,929::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2125::DEBUG::2012-08-30 13:14:28,930::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::DEBUG::2012-08-30 13:14:30,784::task::588::TaskManager.Task::(_updateState) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:14:30,784::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:14:30,784::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`eae0f68f-8cf3-435e-a18e-c21caa227db2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:14:30,784::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:14:30,785::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:14:30,785::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`eae0f68f-8cf3-435e-a18e-c21caa227db2`::Granted request >Thread-104::DEBUG::2012-08-30 13:14:30,785::task::817::TaskManager.Task::(resourceAcquired) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:14:30,786::task::978::TaskManager.Task::(_decref) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:14:30,787::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:14:30,789::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:14:30,790::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:14:30,791::task::1172::TaskManager.Task::(prepare) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:14:30,791::task::588::TaskManager.Task::(_updateState) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:14:30,791::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:14:30,791::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:14:30,791::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:14:30,792::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:14:30,792::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:14:30,792::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:14:30,792::task::978::TaskManager.Task::(_decref) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:14:30,793::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:30,793::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:30,794::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:30,794::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:14:33,264::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:14:33,672::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:35,795::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:38,673::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:38,674::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2128::DEBUG::2012-08-30 13:14:40,568::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2128::DEBUG::2012-08-30 13:14:40,568::task::588::TaskManager.Task::(_updateState) Task=`b50612e2-c1c8-423b-ad5b-54c67dc0cf46`::moving from state init -> state preparing >Thread-2128::INFO::2012-08-30 13:14:40,568::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2128::INFO::2012-08-30 13:14:40,569::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2128::DEBUG::2012-08-30 13:14:40,569::task::1172::TaskManager.Task::(prepare) Task=`b50612e2-c1c8-423b-ad5b-54c67dc0cf46`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2128::DEBUG::2012-08-30 13:14:40,569::task::588::TaskManager.Task::(_updateState) Task=`b50612e2-c1c8-423b-ad5b-54c67dc0cf46`::moving from state preparing -> state finished >Thread-2128::DEBUG::2012-08-30 13:14:40,569::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2128::DEBUG::2012-08-30 13:14:40,570::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2128::DEBUG::2012-08-30 13:14:40,570::task::978::TaskManager.Task::(_decref) Task=`b50612e2-c1c8-423b-ad5b-54c67dc0cf46`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:14:40,796::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:43,675::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2129::DEBUG::2012-08-30 13:14:44,426::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2129::DEBUG::2012-08-30 13:14:44,426::task::588::TaskManager.Task::(_updateState) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::moving from state init -> state preparing >Thread-2129::INFO::2012-08-30 13:14:44,426::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2129::DEBUG::2012-08-30 13:14:44,427::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`db98f15b-e79e-4ffe-99ab-dd1a44e4a99e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2129::DEBUG::2012-08-30 13:14:44,427::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2129::DEBUG::2012-08-30 13:14:44,427::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2129::DEBUG::2012-08-30 13:14:44,427::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`db98f15b-e79e-4ffe-99ab-dd1a44e4a99e`::Granted request >Thread-2129::DEBUG::2012-08-30 13:14:44,428::task::817::TaskManager.Task::(resourceAcquired) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2129::DEBUG::2012-08-30 13:14:44,428::task::978::TaskManager.Task::(_decref) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::ref 1 aborting False >Thread-2129::INFO::2012-08-30 13:14:44,431::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2129::DEBUG::2012-08-30 13:14:44,431::task::1172::TaskManager.Task::(prepare) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2129::DEBUG::2012-08-30 13:14:44,431::task::588::TaskManager.Task::(_updateState) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::moving from state preparing -> state finished >Thread-2129::DEBUG::2012-08-30 13:14:44,432::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2129::DEBUG::2012-08-30 13:14:44,432::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2129::DEBUG::2012-08-30 13:14:44,432::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2129::DEBUG::2012-08-30 13:14:44,432::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2129::DEBUG::2012-08-30 13:14:44,433::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2129::DEBUG::2012-08-30 13:14:44,433::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2129::DEBUG::2012-08-30 13:14:44,433::task::978::TaskManager.Task::(_decref) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:14:45,797::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:45,797::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:48,676::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:50,798::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:14:53,677::task::588::TaskManager.Task::(_updateState) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:14:53,678::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:14:53,678::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`47150741-1c2c-4ec2-a42a-3db704fb7d48`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:14:53,678::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:14:53,679::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:14:53,679::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`47150741-1c2c-4ec2-a42a-3db704fb7d48`::Granted request >Thread-40::DEBUG::2012-08-30 13:14:53,679::task::817::TaskManager.Task::(resourceAcquired) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:14:53,680::task::978::TaskManager.Task::(_decref) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:14:53,681::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:14:53,683::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:14:53,684::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:14:53,685::task::1172::TaskManager.Task::(prepare) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:14:53,685::task::588::TaskManager.Task::(_updateState) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:14:53,685::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:14:53,685::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:14:53,686::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:14:53,686::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:14:53,686::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:14:53,686::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:14:53,687::task::978::TaskManager.Task::(_decref) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:14:53,687::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:53,687::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:53,688::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:14:53,688::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:14:55,799::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:14:56,284::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:14:58,690::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2132::DEBUG::2012-08-30 13:15:00,011::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2132::DEBUG::2012-08-30 13:15:00,011::task::588::TaskManager.Task::(_updateState) Task=`5497a39f-1491-436f-9c60-3e0699cb40b4`::moving from state init -> state preparing >Thread-2132::INFO::2012-08-30 13:15:00,012::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2132::INFO::2012-08-30 13:15:00,012::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2132::DEBUG::2012-08-30 13:15:00,012::task::1172::TaskManager.Task::(prepare) Task=`5497a39f-1491-436f-9c60-3e0699cb40b4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2132::DEBUG::2012-08-30 13:15:00,012::task::588::TaskManager.Task::(_updateState) Task=`5497a39f-1491-436f-9c60-3e0699cb40b4`::moving from state preparing -> state finished >Thread-2132::DEBUG::2012-08-30 13:15:00,013::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2132::DEBUG::2012-08-30 13:15:00,013::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2132::DEBUG::2012-08-30 13:15:00,013::task::978::TaskManager.Task::(_decref) Task=`5497a39f-1491-436f-9c60-3e0699cb40b4`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:15:00,800::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:00,801::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2133::DEBUG::2012-08-30 13:15:03,077::task::588::TaskManager.Task::(_updateState) Task=`ee3b83ef-cd03-47db-992b-b47ff26d4c71`::moving from state init -> state preparing >Thread-2133::INFO::2012-08-30 13:15:03,077::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2133::INFO::2012-08-30 13:15:03,077::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00179195404053', 'lastCheck': 1346303701.100708, 'code': 0, 'valid': True}} >Thread-2133::DEBUG::2012-08-30 13:15:03,078::task::1172::TaskManager.Task::(prepare) Task=`ee3b83ef-cd03-47db-992b-b47ff26d4c71`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00179195404053', 'lastCheck': 1346303701.100708, 'code': 0, 'valid': True}} >Thread-2133::DEBUG::2012-08-30 13:15:03,078::task::588::TaskManager.Task::(_updateState) Task=`ee3b83ef-cd03-47db-992b-b47ff26d4c71`::moving from state preparing -> state finished >Thread-2133::DEBUG::2012-08-30 13:15:03,078::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2133::DEBUG::2012-08-30 13:15:03,078::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2133::DEBUG::2012-08-30 13:15:03,078::task::978::TaskManager.Task::(_decref) Task=`ee3b83ef-cd03-47db-992b-b47ff26d4c71`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:15:03,290::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:15:03,691::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:05,801::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2134::DEBUG::2012-08-30 13:15:06,321::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2134::DEBUG::2012-08-30 13:15:06,321::task::588::TaskManager.Task::(_updateState) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::moving from state init -> state preparing >Thread-2134::INFO::2012-08-30 13:15:06,321::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2134::DEBUG::2012-08-30 13:15:06,322::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a9fc6001-ad1b-4f00-b35c-9a9ff34f984b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2134::DEBUG::2012-08-30 13:15:06,322::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2134::DEBUG::2012-08-30 13:15:06,322::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2134::DEBUG::2012-08-30 13:15:06,322::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a9fc6001-ad1b-4f00-b35c-9a9ff34f984b`::Granted request >Thread-2134::DEBUG::2012-08-30 13:15:06,323::task::817::TaskManager.Task::(resourceAcquired) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2134::DEBUG::2012-08-30 13:15:06,323::task::978::TaskManager.Task::(_decref) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::ref 1 aborting False >Thread-2134::INFO::2012-08-30 13:15:06,326::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2134::DEBUG::2012-08-30 13:15:06,326::task::1172::TaskManager.Task::(prepare) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2134::DEBUG::2012-08-30 13:15:06,326::task::588::TaskManager.Task::(_updateState) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::moving from state preparing -> state finished >Thread-2134::DEBUG::2012-08-30 13:15:06,327::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2134::DEBUG::2012-08-30 13:15:06,327::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2134::DEBUG::2012-08-30 13:15:06,327::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2134::DEBUG::2012-08-30 13:15:06,327::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2134::DEBUG::2012-08-30 13:15:06,328::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2134::DEBUG::2012-08-30 13:15:06,328::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2134::DEBUG::2012-08-30 13:15:06,328::task::978::TaskManager.Task::(_decref) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:15:08,692::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:08,692::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2135::DEBUG::2012-08-30 13:15:09,500::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2135::DEBUG::2012-08-30 13:15:09,501::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:15:10,802::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:13,693::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:15,803::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:15,804::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:18,695::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:20,805::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2137::DEBUG::2012-08-30 13:15:22,116::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2137::DEBUG::2012-08-30 13:15:22,117::task::588::TaskManager.Task::(_updateState) Task=`28a02c4e-b0ae-40a0-815a-85e572874024`::moving from state init -> state preparing >Thread-2137::INFO::2012-08-30 13:15:22,117::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2137::INFO::2012-08-30 13:15:22,117::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2137::DEBUG::2012-08-30 13:15:22,117::task::1172::TaskManager.Task::(prepare) Task=`28a02c4e-b0ae-40a0-815a-85e572874024`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2137::DEBUG::2012-08-30 13:15:22,118::task::588::TaskManager.Task::(_updateState) Task=`28a02c4e-b0ae-40a0-815a-85e572874024`::moving from state preparing -> state finished >Thread-2137::DEBUG::2012-08-30 13:15:22,118::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2137::DEBUG::2012-08-30 13:15:22,118::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2137::DEBUG::2012-08-30 13:15:22,118::task::978::TaskManager.Task::(_decref) Task=`28a02c4e-b0ae-40a0-815a-85e572874024`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:15:23,696::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:23,697::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:25,806::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:15:26,309::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2139::DEBUG::2012-08-30 13:15:28,339::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2139::DEBUG::2012-08-30 13:15:28,340::task::588::TaskManager.Task::(_updateState) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::moving from state init -> state preparing >Thread-2139::INFO::2012-08-30 13:15:28,340::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2139::DEBUG::2012-08-30 13:15:28,340::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0724596-2461-4844-a836-23678d44f294`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2139::DEBUG::2012-08-30 13:15:28,341::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2139::DEBUG::2012-08-30 13:15:28,341::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2139::DEBUG::2012-08-30 13:15:28,341::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0724596-2461-4844-a836-23678d44f294`::Granted request >Thread-2139::DEBUG::2012-08-30 13:15:28,342::task::817::TaskManager.Task::(resourceAcquired) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2139::DEBUG::2012-08-30 13:15:28,342::task::978::TaskManager.Task::(_decref) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::ref 1 aborting False >Thread-2139::INFO::2012-08-30 13:15:28,345::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2139::DEBUG::2012-08-30 13:15:28,345::task::1172::TaskManager.Task::(prepare) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2139::DEBUG::2012-08-30 13:15:28,345::task::588::TaskManager.Task::(_updateState) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::moving from state preparing -> state finished >Thread-2139::DEBUG::2012-08-30 13:15:28,345::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2139::DEBUG::2012-08-30 13:15:28,346::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2139::DEBUG::2012-08-30 13:15:28,346::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2139::DEBUG::2012-08-30 13:15:28,346::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2139::DEBUG::2012-08-30 13:15:28,346::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2139::DEBUG::2012-08-30 13:15:28,347::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2139::DEBUG::2012-08-30 13:15:28,347::task::978::TaskManager.Task::(_decref) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:15:28,698::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:15:30,807::task::588::TaskManager.Task::(_updateState) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:15:30,807::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:15:30,808::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c168573c-0160-409e-8bac-5e6c60efc047`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:15:30,808::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:15:30,808::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:15:30,808::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c168573c-0160-409e-8bac-5e6c60efc047`::Granted request >Thread-104::DEBUG::2012-08-30 13:15:30,809::task::817::TaskManager.Task::(resourceAcquired) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:15:30,809::task::978::TaskManager.Task::(_decref) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:15:30,810::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:15:30,813::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:15:30,814::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:15:30,814::task::1172::TaskManager.Task::(prepare) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:15:30,814::task::588::TaskManager.Task::(_updateState) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:15:30,814::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:15:30,815::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:15:30,815::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:15:30,815::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:15:30,815::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:15:30,816::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:15:30,816::task::978::TaskManager.Task::(_decref) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:15:30,816::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:30,817::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:30,817::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:30,817::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-26::DEBUG::2012-08-30 13:15:31,113::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:15:31,113::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-26::DEBUG::2012-08-30 13:15:31,114::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None) >Thread-26::DEBUG::2012-08-30 13:15:31,222::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5 >Thread-26::WARNING::2012-08-30 13:15:31,223::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found'] >Thread-26::DEBUG::2012-08-30 13:15:31,223::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-26::DEBUG::2012-08-30 13:15:31,228::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:15:31,229::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend >Thread-26::DEBUG::2012-08-30 13:15:31,231::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d'] >Thread-26::DEBUG::2012-08-30 13:15:31,232::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-26::WARNING::2012-08-30 13:15:31,233::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered >Thread-26::WARNING::2012-08-30 13:15:31,233::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered >VM Channels Listener::DEBUG::2012-08-30 13:15:33,317::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:15:33,700::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:35,818::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:38,701::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:38,702::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:40,819::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2147::DEBUG::2012-08-30 13:15:43,119::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2147::DEBUG::2012-08-30 13:15:43,119::task::588::TaskManager.Task::(_updateState) Task=`0d397ba6-4b4b-4496-944b-af34443256b2`::moving from state init -> state preparing >Thread-2147::INFO::2012-08-30 13:15:43,119::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2147::INFO::2012-08-30 13:15:43,120::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2147::DEBUG::2012-08-30 13:15:43,120::task::1172::TaskManager.Task::(prepare) Task=`0d397ba6-4b4b-4496-944b-af34443256b2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2147::DEBUG::2012-08-30 13:15:43,120::task::588::TaskManager.Task::(_updateState) Task=`0d397ba6-4b4b-4496-944b-af34443256b2`::moving from state preparing -> state finished >Thread-2147::DEBUG::2012-08-30 13:15:43,121::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2147::DEBUG::2012-08-30 13:15:43,121::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2147::DEBUG::2012-08-30 13:15:43,121::task::978::TaskManager.Task::(_decref) Task=`0d397ba6-4b4b-4496-944b-af34443256b2`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:15:43,703::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2148::DEBUG::2012-08-30 13:15:45,097::task::588::TaskManager.Task::(_updateState) Task=`8193a844-30c0-40a8-af49-f45f1675d169`::moving from state init -> state preparing >Thread-2148::INFO::2012-08-30 13:15:45,098::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2148::INFO::2012-08-30 13:15:45,098::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163698196411', 'lastCheck': 1346303741.2457759, 'code': 0, 'valid': True}} >Thread-2148::DEBUG::2012-08-30 13:15:45,098::task::1172::TaskManager.Task::(prepare) Task=`8193a844-30c0-40a8-af49-f45f1675d169`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163698196411', 'lastCheck': 1346303741.2457759, 'code': 0, 'valid': True}} >Thread-2148::DEBUG::2012-08-30 13:15:45,098::task::588::TaskManager.Task::(_updateState) Task=`8193a844-30c0-40a8-af49-f45f1675d169`::moving from state preparing -> state finished >Thread-2148::DEBUG::2012-08-30 13:15:45,099::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2148::DEBUG::2012-08-30 13:15:45,099::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2148::DEBUG::2012-08-30 13:15:45,099::task::978::TaskManager.Task::(_decref) Task=`8193a844-30c0-40a8-af49-f45f1675d169`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:15:45,820::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:45,821::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2149::DEBUG::2012-08-30 13:15:47,201::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2149::DEBUG::2012-08-30 13:15:47,202::task::588::TaskManager.Task::(_updateState) Task=`76963a33-87ef-4caa-9455-616dd605360a`::moving from state init -> state preparing >Thread-2149::INFO::2012-08-30 13:15:47,202::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2149::DEBUG::2012-08-30 13:15:47,202::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`53d8b6d8-1e05-4696-9c07-1a825bc44616`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2149::DEBUG::2012-08-30 13:15:47,203::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2149::DEBUG::2012-08-30 13:15:47,203::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2149::DEBUG::2012-08-30 13:15:47,203::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`53d8b6d8-1e05-4696-9c07-1a825bc44616`::Granted request >Thread-2149::DEBUG::2012-08-30 13:15:47,204::task::817::TaskManager.Task::(resourceAcquired) Task=`76963a33-87ef-4caa-9455-616dd605360a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2149::DEBUG::2012-08-30 13:15:47,204::task::978::TaskManager.Task::(_decref) Task=`76963a33-87ef-4caa-9455-616dd605360a`::ref 1 aborting False >Thread-2149::INFO::2012-08-30 13:15:47,207::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2149::DEBUG::2012-08-30 13:15:47,207::task::1172::TaskManager.Task::(prepare) Task=`76963a33-87ef-4caa-9455-616dd605360a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2149::DEBUG::2012-08-30 13:15:47,207::task::588::TaskManager.Task::(_updateState) Task=`76963a33-87ef-4caa-9455-616dd605360a`::moving from state preparing -> state finished >Thread-2149::DEBUG::2012-08-30 13:15:47,207::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2149::DEBUG::2012-08-30 13:15:47,208::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2149::DEBUG::2012-08-30 13:15:47,208::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2149::DEBUG::2012-08-30 13:15:47,208::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2149::DEBUG::2012-08-30 13:15:47,208::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2149::DEBUG::2012-08-30 13:15:47,209::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2149::DEBUG::2012-08-30 13:15:47,209::task::978::TaskManager.Task::(_decref) Task=`76963a33-87ef-4caa-9455-616dd605360a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:15:48,704::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2150::DEBUG::2012-08-30 13:15:49,099::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2150::DEBUG::2012-08-30 13:15:49,099::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:15:50,822::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:15:53,705::task::588::TaskManager.Task::(_updateState) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:15:53,706::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:15:53,706::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e5bc4a05-b145-4a3a-89bd-39f424f0ddb3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:15:53,706::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:15:53,707::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:15:53,707::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e5bc4a05-b145-4a3a-89bd-39f424f0ddb3`::Granted request >Thread-40::DEBUG::2012-08-30 13:15:53,707::task::817::TaskManager.Task::(resourceAcquired) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:15:53,708::task::978::TaskManager.Task::(_decref) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:15:53,709::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:15:53,711::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:15:53,713::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:15:53,713::task::1172::TaskManager.Task::(prepare) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:15:53,713::task::588::TaskManager.Task::(_updateState) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:15:53,713::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:15:53,713::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:15:53,714::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:15:53,714::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:15:53,714::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:15:53,714::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:15:53,715::task::978::TaskManager.Task::(_decref) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:15:53,715::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:53,716::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:53,716::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:15:53,716::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:15:55,823::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:15:56,337::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:15:58,718::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:00,824::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:00,824::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2152::DEBUG::2012-08-30 13:16:01,190::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2152::DEBUG::2012-08-30 13:16:01,191::task::588::TaskManager.Task::(_updateState) Task=`ac7e0c36-b5c3-4e19-934f-001cd5f0b7c5`::moving from state init -> state preparing >Thread-2152::INFO::2012-08-30 13:16:01,191::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2152::INFO::2012-08-30 13:16:01,192::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2152::DEBUG::2012-08-30 13:16:01,192::task::1172::TaskManager.Task::(prepare) Task=`ac7e0c36-b5c3-4e19-934f-001cd5f0b7c5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2152::DEBUG::2012-08-30 13:16:01,192::task::588::TaskManager.Task::(_updateState) Task=`ac7e0c36-b5c3-4e19-934f-001cd5f0b7c5`::moving from state preparing -> state finished >Thread-2152::DEBUG::2012-08-30 13:16:01,192::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2152::DEBUG::2012-08-30 13:16:01,193::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2152::DEBUG::2012-08-30 13:16:01,193::task::978::TaskManager.Task::(_decref) Task=`ac7e0c36-b5c3-4e19-934f-001cd5f0b7c5`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:16:03,343::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:16:03,719::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2154::DEBUG::2012-08-30 13:16:05,082::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2154::DEBUG::2012-08-30 13:16:05,082::task::588::TaskManager.Task::(_updateState) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::moving from state init -> state preparing >Thread-2154::INFO::2012-08-30 13:16:05,082::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2154::DEBUG::2012-08-30 13:16:05,083::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f28ca09c-02e2-4518-91e8-54ce54ad2312`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2154::DEBUG::2012-08-30 13:16:05,083::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2154::DEBUG::2012-08-30 13:16:05,083::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2154::DEBUG::2012-08-30 13:16:05,083::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f28ca09c-02e2-4518-91e8-54ce54ad2312`::Granted request >Thread-2154::DEBUG::2012-08-30 13:16:05,084::task::817::TaskManager.Task::(resourceAcquired) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2154::DEBUG::2012-08-30 13:16:05,084::task::978::TaskManager.Task::(_decref) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::ref 1 aborting False >Thread-2154::INFO::2012-08-30 13:16:05,087::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2154::DEBUG::2012-08-30 13:16:05,087::task::1172::TaskManager.Task::(prepare) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2154::DEBUG::2012-08-30 13:16:05,087::task::588::TaskManager.Task::(_updateState) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::moving from state preparing -> state finished >Thread-2154::DEBUG::2012-08-30 13:16:05,088::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2154::DEBUG::2012-08-30 13:16:05,088::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2154::DEBUG::2012-08-30 13:16:05,088::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2154::DEBUG::2012-08-30 13:16:05,088::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2154::DEBUG::2012-08-30 13:16:05,089::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2154::DEBUG::2012-08-30 13:16:05,089::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2154::DEBUG::2012-08-30 13:16:05,089::task::978::TaskManager.Task::(_decref) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:16:05,825::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:08,720::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:08,721::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:10,826::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:13,722::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:15,827::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:15,827::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2157::DEBUG::2012-08-30 13:16:17,448::task::588::TaskManager.Task::(_updateState) Task=`81bee071-6cb5-4bfb-8edb-50ed12ffdffd`::moving from state init -> state preparing >Thread-2157::INFO::2012-08-30 13:16:17,448::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2157::INFO::2012-08-30 13:16:17,448::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346303771.2661681, 'code': 0, 'valid': True}} >Thread-2157::DEBUG::2012-08-30 13:16:17,449::task::1172::TaskManager.Task::(prepare) Task=`81bee071-6cb5-4bfb-8edb-50ed12ffdffd`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346303771.2661681, 'code': 0, 'valid': True}} >Thread-2157::DEBUG::2012-08-30 13:16:17,449::task::588::TaskManager.Task::(_updateState) Task=`81bee071-6cb5-4bfb-8edb-50ed12ffdffd`::moving from state preparing -> state finished >Thread-2157::DEBUG::2012-08-30 13:16:17,449::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2157::DEBUG::2012-08-30 13:16:17,449::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2157::DEBUG::2012-08-30 13:16:17,450::task::978::TaskManager.Task::(_decref) Task=`81bee071-6cb5-4bfb-8edb-50ed12ffdffd`::ref 0 aborting False >Thread-2158::DEBUG::2012-08-30 13:16:18,404::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2158::DEBUG::2012-08-30 13:16:18,405::task::588::TaskManager.Task::(_updateState) Task=`067b857b-384c-49a3-877f-12c21788f436`::moving from state init -> state preparing >Thread-2158::INFO::2012-08-30 13:16:18,405::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2158::INFO::2012-08-30 13:16:18,405::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2158::DEBUG::2012-08-30 13:16:18,406::task::1172::TaskManager.Task::(prepare) Task=`067b857b-384c-49a3-877f-12c21788f436`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2158::DEBUG::2012-08-30 13:16:18,406::task::588::TaskManager.Task::(_updateState) Task=`067b857b-384c-49a3-877f-12c21788f436`::moving from state preparing -> state finished >Thread-2158::DEBUG::2012-08-30 13:16:18,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2158::DEBUG::2012-08-30 13:16:18,406::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2158::DEBUG::2012-08-30 13:16:18,407::task::978::TaskManager.Task::(_decref) Task=`067b857b-384c-49a3-877f-12c21788f436`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:16:18,723::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2159::DEBUG::2012-08-30 13:16:20,586::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2159::DEBUG::2012-08-30 13:16:20,587::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:16:20,828::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2160::DEBUG::2012-08-30 13:16:23,583::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2160::DEBUG::2012-08-30 13:16:23,584::task::588::TaskManager.Task::(_updateState) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::moving from state init -> state preparing >Thread-2160::INFO::2012-08-30 13:16:23,584::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2160::DEBUG::2012-08-30 13:16:23,584::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2cd7321d-4daf-4df0-be90-5f6128d0c560`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2160::DEBUG::2012-08-30 13:16:23,585::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2160::DEBUG::2012-08-30 13:16:23,585::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2160::DEBUG::2012-08-30 13:16:23,585::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2cd7321d-4daf-4df0-be90-5f6128d0c560`::Granted request >Thread-2160::DEBUG::2012-08-30 13:16:23,585::task::817::TaskManager.Task::(resourceAcquired) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2160::DEBUG::2012-08-30 13:16:23,586::task::978::TaskManager.Task::(_decref) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::ref 1 aborting False >Thread-2160::INFO::2012-08-30 13:16:23,589::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2160::DEBUG::2012-08-30 13:16:23,589::task::1172::TaskManager.Task::(prepare) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2160::DEBUG::2012-08-30 13:16:23,589::task::588::TaskManager.Task::(_updateState) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::moving from state preparing -> state finished >Thread-2160::DEBUG::2012-08-30 13:16:23,589::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2160::DEBUG::2012-08-30 13:16:23,590::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2160::DEBUG::2012-08-30 13:16:23,590::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2160::DEBUG::2012-08-30 13:16:23,590::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2160::DEBUG::2012-08-30 13:16:23,590::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2160::DEBUG::2012-08-30 13:16:23,591::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2160::DEBUG::2012-08-30 13:16:23,591::task::978::TaskManager.Task::(_decref) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:16:23,724::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:23,725::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:25,829::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:16:26,362::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:16:28,726::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:16:30,830::task::588::TaskManager.Task::(_updateState) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:16:30,830::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:16:30,831::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`269ab286-3a0f-45e1-93f4-184f8a7b808c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:16:30,831::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:16:30,831::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:16:30,831::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`269ab286-3a0f-45e1-93f4-184f8a7b808c`::Granted request >Thread-104::DEBUG::2012-08-30 13:16:30,832::task::817::TaskManager.Task::(resourceAcquired) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:16:30,832::task::978::TaskManager.Task::(_decref) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:16:30,834::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:16:30,836::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:16:30,837::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:16:30,837::task::1172::TaskManager.Task::(prepare) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:16:30,837::task::588::TaskManager.Task::(_updateState) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:16:30,838::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:16:30,838::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:16:30,838::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:16:30,838::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:16:30,839::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:16:30,839::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:16:30,839::task::978::TaskManager.Task::(_decref) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:16:30,839::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:30,840::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:30,840::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:30,841::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:16:33,366::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:16:33,727::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:35,842::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2163::DEBUG::2012-08-30 13:16:37,708::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2163::DEBUG::2012-08-30 13:16:37,708::task::588::TaskManager.Task::(_updateState) Task=`58a13bbc-4625-4e73-a8d5-72da1f8f0443`::moving from state init -> state preparing >Thread-2163::INFO::2012-08-30 13:16:37,709::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2163::INFO::2012-08-30 13:16:37,709::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2163::DEBUG::2012-08-30 13:16:37,709::task::1172::TaskManager.Task::(prepare) Task=`58a13bbc-4625-4e73-a8d5-72da1f8f0443`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2163::DEBUG::2012-08-30 13:16:37,709::task::588::TaskManager.Task::(_updateState) Task=`58a13bbc-4625-4e73-a8d5-72da1f8f0443`::moving from state preparing -> state finished >Thread-2163::DEBUG::2012-08-30 13:16:37,710::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2163::DEBUG::2012-08-30 13:16:37,710::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2163::DEBUG::2012-08-30 13:16:37,710::task::978::TaskManager.Task::(_decref) Task=`58a13bbc-4625-4e73-a8d5-72da1f8f0443`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:16:38,728::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:38,729::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:40,842::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2165::DEBUG::2012-08-30 13:16:42,494::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2165::DEBUG::2012-08-30 13:16:42,495::task::588::TaskManager.Task::(_updateState) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::moving from state init -> state preparing >Thread-2165::INFO::2012-08-30 13:16:42,495::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2165::DEBUG::2012-08-30 13:16:42,496::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`87f1bfd4-e1af-4160-8261-0345e4124b7c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2165::DEBUG::2012-08-30 13:16:42,496::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2165::DEBUG::2012-08-30 13:16:42,496::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2165::DEBUG::2012-08-30 13:16:42,496::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`87f1bfd4-e1af-4160-8261-0345e4124b7c`::Granted request >Thread-2165::DEBUG::2012-08-30 13:16:42,497::task::817::TaskManager.Task::(resourceAcquired) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2165::DEBUG::2012-08-30 13:16:42,497::task::978::TaskManager.Task::(_decref) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::ref 1 aborting False >Thread-2165::INFO::2012-08-30 13:16:42,500::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2165::DEBUG::2012-08-30 13:16:42,500::task::1172::TaskManager.Task::(prepare) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2165::DEBUG::2012-08-30 13:16:42,500::task::588::TaskManager.Task::(_updateState) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::moving from state preparing -> state finished >Thread-2165::DEBUG::2012-08-30 13:16:42,501::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2165::DEBUG::2012-08-30 13:16:42,501::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2165::DEBUG::2012-08-30 13:16:42,501::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2165::DEBUG::2012-08-30 13:16:42,501::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2165::DEBUG::2012-08-30 13:16:42,502::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2165::DEBUG::2012-08-30 13:16:42,502::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2165::DEBUG::2012-08-30 13:16:42,502::task::978::TaskManager.Task::(_decref) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:16:43,730::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:45,843::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:45,844::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:48,731::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:16:50,845::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:16:53,732::task::588::TaskManager.Task::(_updateState) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:16:53,733::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:16:53,733::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`24b659ec-30a1-45e0-aba6-4eeddebc09d4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:16:53,734::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:16:53,734::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:16:53,734::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`24b659ec-30a1-45e0-aba6-4eeddebc09d4`::Granted request >Thread-40::DEBUG::2012-08-30 13:16:53,735::task::817::TaskManager.Task::(resourceAcquired) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:16:53,735::task::978::TaskManager.Task::(_decref) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:16:53,737::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:16:53,739::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:16:53,740::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:16:53,740::task::1172::TaskManager.Task::(prepare) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:16:53,740::task::588::TaskManager.Task::(_updateState) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:16:53,740::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:16:53,741::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:16:53,741::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:16:53,741::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:16:53,741::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:16:53,742::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:16:53,742::task::978::TaskManager.Task::(_decref) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:16:53,742::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:53,743::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:53,743::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:16:53,744::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2167::DEBUG::2012-08-30 13:16:55,046::task::588::TaskManager.Task::(_updateState) Task=`6cd31a34-05c9-4b97-82fb-eb367510df25`::moving from state init -> state preparing >Thread-2167::INFO::2012-08-30 13:16:55,046::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2167::INFO::2012-08-30 13:16:55,046::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178194046021', 'lastCheck': 1346303811.29179, 'code': 0, 'valid': True}} >Thread-2167::DEBUG::2012-08-30 13:16:55,047::task::1172::TaskManager.Task::(prepare) Task=`6cd31a34-05c9-4b97-82fb-eb367510df25`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178194046021', 'lastCheck': 1346303811.29179, 'code': 0, 'valid': True}} >Thread-2167::DEBUG::2012-08-30 13:16:55,047::task::588::TaskManager.Task::(_updateState) Task=`6cd31a34-05c9-4b97-82fb-eb367510df25`::moving from state preparing -> state finished >Thread-2167::DEBUG::2012-08-30 13:16:55,047::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2167::DEBUG::2012-08-30 13:16:55,047::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2167::DEBUG::2012-08-30 13:16:55,048::task::978::TaskManager.Task::(_decref) Task=`6cd31a34-05c9-4b97-82fb-eb367510df25`::ref 0 aborting False >Thread-2168::DEBUG::2012-08-30 13:16:55,049::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2168::DEBUG::2012-08-30 13:16:55,050::task::588::TaskManager.Task::(_updateState) Task=`9ca27cd1-f236-4d7d-a8b0-7ad32c82dd4e`::moving from state init -> state preparing >Thread-2168::INFO::2012-08-30 13:16:55,050::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2168::INFO::2012-08-30 13:16:55,050::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2168::DEBUG::2012-08-30 13:16:55,051::task::1172::TaskManager.Task::(prepare) Task=`9ca27cd1-f236-4d7d-a8b0-7ad32c82dd4e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2168::DEBUG::2012-08-30 13:16:55,051::task::588::TaskManager.Task::(_updateState) Task=`9ca27cd1-f236-4d7d-a8b0-7ad32c82dd4e`::moving from state preparing -> state finished >Thread-2168::DEBUG::2012-08-30 13:16:55,051::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2168::DEBUG::2012-08-30 13:16:55,051::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2168::DEBUG::2012-08-30 13:16:55,051::task::978::TaskManager.Task::(_decref) Task=`9ca27cd1-f236-4d7d-a8b0-7ad32c82dd4e`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:16:55,846::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:16:56,387::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2169::DEBUG::2012-08-30 13:16:58,191::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2170::DEBUG::2012-08-30 13:16:58,191::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2169::DEBUG::2012-08-30 13:16:58,192::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-2170::DEBUG::2012-08-30 13:16:58,192::task::588::TaskManager.Task::(_updateState) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::moving from state init -> state preparing >Thread-2170::INFO::2012-08-30 13:16:58,193::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2170::DEBUG::2012-08-30 13:16:58,194::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c7ef120e-c4e4-41b7-9806-fa3dc35e2a21`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2170::DEBUG::2012-08-30 13:16:58,194::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2170::DEBUG::2012-08-30 13:16:58,194::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2170::DEBUG::2012-08-30 13:16:58,195::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c7ef120e-c4e4-41b7-9806-fa3dc35e2a21`::Granted request >Thread-2170::DEBUG::2012-08-30 13:16:58,195::task::817::TaskManager.Task::(resourceAcquired) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2170::DEBUG::2012-08-30 13:16:58,195::task::978::TaskManager.Task::(_decref) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::ref 1 aborting False >Thread-2170::INFO::2012-08-30 13:16:58,198::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2170::DEBUG::2012-08-30 13:16:58,198::task::1172::TaskManager.Task::(prepare) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2170::DEBUG::2012-08-30 13:16:58,199::task::588::TaskManager.Task::(_updateState) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::moving from state preparing -> state finished >Thread-2170::DEBUG::2012-08-30 13:16:58,199::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2170::DEBUG::2012-08-30 13:16:58,199::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2170::DEBUG::2012-08-30 13:16:58,199::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2170::DEBUG::2012-08-30 13:16:58,200::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2170::DEBUG::2012-08-30 13:16:58,200::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2170::DEBUG::2012-08-30 13:16:58,200::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2170::DEBUG::2012-08-30 13:16:58,200::task::978::TaskManager.Task::(_decref) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:16:58,745::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:00,847::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:00,847::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:17:03,393::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:17:03,746::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:05,848::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:08,747::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:08,748::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:10,849::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2173::DEBUG::2012-08-30 13:17:11,439::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2173::DEBUG::2012-08-30 13:17:11,439::task::588::TaskManager.Task::(_updateState) Task=`2638c9bb-7364-4483-8bf9-bfdbd2dddc43`::moving from state init -> state preparing >Thread-2173::INFO::2012-08-30 13:17:11,440::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2173::INFO::2012-08-30 13:17:11,440::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2173::DEBUG::2012-08-30 13:17:11,440::task::1172::TaskManager.Task::(prepare) Task=`2638c9bb-7364-4483-8bf9-bfdbd2dddc43`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2173::DEBUG::2012-08-30 13:17:11,440::task::588::TaskManager.Task::(_updateState) Task=`2638c9bb-7364-4483-8bf9-bfdbd2dddc43`::moving from state preparing -> state finished >Thread-2173::DEBUG::2012-08-30 13:17:11,440::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2173::DEBUG::2012-08-30 13:17:11,441::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2173::DEBUG::2012-08-30 13:17:11,441::task::978::TaskManager.Task::(_decref) Task=`2638c9bb-7364-4483-8bf9-bfdbd2dddc43`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:17:13,749::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2175::DEBUG::2012-08-30 13:17:14,662::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2175::DEBUG::2012-08-30 13:17:14,662::task::588::TaskManager.Task::(_updateState) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::moving from state init -> state preparing >Thread-2175::INFO::2012-08-30 13:17:14,663::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2175::DEBUG::2012-08-30 13:17:14,663::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`139bf660-d5eb-4bb4-9f89-b603e0812e74`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2175::DEBUG::2012-08-30 13:17:14,663::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2175::DEBUG::2012-08-30 13:17:14,664::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2175::DEBUG::2012-08-30 13:17:14,664::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`139bf660-d5eb-4bb4-9f89-b603e0812e74`::Granted request >Thread-2175::DEBUG::2012-08-30 13:17:14,664::task::817::TaskManager.Task::(resourceAcquired) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2175::DEBUG::2012-08-30 13:17:14,664::task::978::TaskManager.Task::(_decref) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::ref 1 aborting False >Thread-2175::INFO::2012-08-30 13:17:14,667::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2175::DEBUG::2012-08-30 13:17:14,668::task::1172::TaskManager.Task::(prepare) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2175::DEBUG::2012-08-30 13:17:14,668::task::588::TaskManager.Task::(_updateState) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::moving from state preparing -> state finished >Thread-2175::DEBUG::2012-08-30 13:17:14,668::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2175::DEBUG::2012-08-30 13:17:14,668::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2175::DEBUG::2012-08-30 13:17:14,669::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2175::DEBUG::2012-08-30 13:17:14,669::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2175::DEBUG::2012-08-30 13:17:14,669::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2175::DEBUG::2012-08-30 13:17:14,669::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2175::DEBUG::2012-08-30 13:17:14,670::task::978::TaskManager.Task::(_decref) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:17:15,850::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:15,851::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:18,750::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:20,851::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:23,752::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:23,752::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:25,852::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:17:26,411::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2177::DEBUG::2012-08-30 13:17:27,479::task::588::TaskManager.Task::(_updateState) Task=`de954292-7921-4f36-8b2d-30c84dcb05ec`::moving from state init -> state preparing >Thread-2177::INFO::2012-08-30 13:17:27,479::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2177::INFO::2012-08-30 13:17:27,480::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303841.311954, 'code': 0, 'valid': True}} >Thread-2177::DEBUG::2012-08-30 13:17:27,480::task::1172::TaskManager.Task::(prepare) Task=`de954292-7921-4f36-8b2d-30c84dcb05ec`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303841.311954, 'code': 0, 'valid': True}} >Thread-2177::DEBUG::2012-08-30 13:17:27,480::task::588::TaskManager.Task::(_updateState) Task=`de954292-7921-4f36-8b2d-30c84dcb05ec`::moving from state preparing -> state finished >Thread-2177::DEBUG::2012-08-30 13:17:27,480::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2177::DEBUG::2012-08-30 13:17:27,481::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2177::DEBUG::2012-08-30 13:17:27,481::task::978::TaskManager.Task::(_decref) Task=`de954292-7921-4f36-8b2d-30c84dcb05ec`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:17:28,754::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2178::DEBUG::2012-08-30 13:17:29,443::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2178::DEBUG::2012-08-30 13:17:29,443::task::588::TaskManager.Task::(_updateState) Task=`dce3bbc0-15c3-4d76-9c4f-b0c2d2cd77f7`::moving from state init -> state preparing >Thread-2178::INFO::2012-08-30 13:17:29,443::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2178::INFO::2012-08-30 13:17:29,444::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2178::DEBUG::2012-08-30 13:17:29,444::task::1172::TaskManager.Task::(prepare) Task=`dce3bbc0-15c3-4d76-9c4f-b0c2d2cd77f7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2178::DEBUG::2012-08-30 13:17:29,444::task::588::TaskManager.Task::(_updateState) Task=`dce3bbc0-15c3-4d76-9c4f-b0c2d2cd77f7`::moving from state preparing -> state finished >Thread-2178::DEBUG::2012-08-30 13:17:29,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2178::DEBUG::2012-08-30 13:17:29,445::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2178::DEBUG::2012-08-30 13:17:29,445::task::978::TaskManager.Task::(_decref) Task=`dce3bbc0-15c3-4d76-9c4f-b0c2d2cd77f7`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 13:17:30,853::task::588::TaskManager.Task::(_updateState) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:17:30,854::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:17:30,854::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`5e479691-3c05-415c-a345-977919d4a585`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:17:30,854::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:17:30,855::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:17:30,855::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`5e479691-3c05-415c-a345-977919d4a585`::Granted request >Thread-104::DEBUG::2012-08-30 13:17:30,855::task::817::TaskManager.Task::(resourceAcquired) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:17:30,855::task::978::TaskManager.Task::(_decref) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:17:30,857::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:17:30,859::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:17:30,860::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:17:30,860::task::1172::TaskManager.Task::(prepare) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:17:30,861::task::588::TaskManager.Task::(_updateState) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:17:30,861::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:17:30,861::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:17:30,861::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:17:30,862::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:17:30,862::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:17:30,862::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:17:30,862::task::978::TaskManager.Task::(_decref) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:17:30,863::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:30,863::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:30,864::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:30,864::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2179::DEBUG::2012-08-30 13:17:31,092::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2179::DEBUG::2012-08-30 13:17:31,093::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-2180::DEBUG::2012-08-30 13:17:32,768::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2180::DEBUG::2012-08-30 13:17:32,768::task::588::TaskManager.Task::(_updateState) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::moving from state init -> state preparing >Thread-2180::INFO::2012-08-30 13:17:32,768::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2180::DEBUG::2012-08-30 13:17:32,769::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`bff1fac9-1131-453e-b37c-f67087828d0e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2180::DEBUG::2012-08-30 13:17:32,769::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2180::DEBUG::2012-08-30 13:17:32,769::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2180::DEBUG::2012-08-30 13:17:32,770::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`bff1fac9-1131-453e-b37c-f67087828d0e`::Granted request >Thread-2180::DEBUG::2012-08-30 13:17:32,770::task::817::TaskManager.Task::(resourceAcquired) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2180::DEBUG::2012-08-30 13:17:32,770::task::978::TaskManager.Task::(_decref) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::ref 1 aborting False >Thread-2180::INFO::2012-08-30 13:17:32,773::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2180::DEBUG::2012-08-30 13:17:32,773::task::1172::TaskManager.Task::(prepare) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2180::DEBUG::2012-08-30 13:17:32,774::task::588::TaskManager.Task::(_updateState) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::moving from state preparing -> state finished >Thread-2180::DEBUG::2012-08-30 13:17:32,774::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2180::DEBUG::2012-08-30 13:17:32,774::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2180::DEBUG::2012-08-30 13:17:32,774::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2180::DEBUG::2012-08-30 13:17:32,775::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2180::DEBUG::2012-08-30 13:17:32,775::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2180::DEBUG::2012-08-30 13:17:32,775::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2180::DEBUG::2012-08-30 13:17:32,775::task::978::TaskManager.Task::(_decref) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:17:33,417::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:17:33,755::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:35,865::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:38,756::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:38,757::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:40,866::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:43,758::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2183::DEBUG::2012-08-30 13:17:45,493::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2183::DEBUG::2012-08-30 13:17:45,494::task::588::TaskManager.Task::(_updateState) Task=`e9ea8caa-5134-41be-a3d4-615cb75efd8c`::moving from state init -> state preparing >Thread-2183::INFO::2012-08-30 13:17:45,494::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2183::INFO::2012-08-30 13:17:45,494::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2183::DEBUG::2012-08-30 13:17:45,494::task::1172::TaskManager.Task::(prepare) Task=`e9ea8caa-5134-41be-a3d4-615cb75efd8c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2183::DEBUG::2012-08-30 13:17:45,495::task::588::TaskManager.Task::(_updateState) Task=`e9ea8caa-5134-41be-a3d4-615cb75efd8c`::moving from state preparing -> state finished >Thread-2183::DEBUG::2012-08-30 13:17:45,495::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2183::DEBUG::2012-08-30 13:17:45,495::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2183::DEBUG::2012-08-30 13:17:45,495::task::978::TaskManager.Task::(_decref) Task=`e9ea8caa-5134-41be-a3d4-615cb75efd8c`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:17:45,867::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:45,867::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2185::DEBUG::2012-08-30 13:17:48,714::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2185::DEBUG::2012-08-30 13:17:48,714::task::588::TaskManager.Task::(_updateState) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::moving from state init -> state preparing >Thread-2185::INFO::2012-08-30 13:17:48,715::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2185::DEBUG::2012-08-30 13:17:48,715::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a78b4188-2e5c-4e23-bef6-c34002715659`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2185::DEBUG::2012-08-30 13:17:48,715::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2185::DEBUG::2012-08-30 13:17:48,716::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2185::DEBUG::2012-08-30 13:17:48,716::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a78b4188-2e5c-4e23-bef6-c34002715659`::Granted request >Thread-2185::DEBUG::2012-08-30 13:17:48,716::task::817::TaskManager.Task::(resourceAcquired) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2185::DEBUG::2012-08-30 13:17:48,716::task::978::TaskManager.Task::(_decref) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::ref 1 aborting False >Thread-2185::INFO::2012-08-30 13:17:48,719::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2185::DEBUG::2012-08-30 13:17:48,720::task::1172::TaskManager.Task::(prepare) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2185::DEBUG::2012-08-30 13:17:48,720::task::588::TaskManager.Task::(_updateState) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::moving from state preparing -> state finished >Thread-2185::DEBUG::2012-08-30 13:17:48,720::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2185::DEBUG::2012-08-30 13:17:48,720::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2185::DEBUG::2012-08-30 13:17:48,721::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2185::DEBUG::2012-08-30 13:17:48,721::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2185::DEBUG::2012-08-30 13:17:48,721::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2185::DEBUG::2012-08-30 13:17:48,721::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2185::DEBUG::2012-08-30 13:17:48,722::task::978::TaskManager.Task::(_decref) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:17:48,759::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:50,868::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:17:53,760::task::588::TaskManager.Task::(_updateState) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:17:53,760::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:17:53,761::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c8bcf974-c591-44fb-b238-84bd02425ea2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:17:53,761::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:17:53,762::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:17:53,762::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c8bcf974-c591-44fb-b238-84bd02425ea2`::Granted request >Thread-40::DEBUG::2012-08-30 13:17:53,762::task::817::TaskManager.Task::(resourceAcquired) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:17:53,762::task::978::TaskManager.Task::(_decref) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:17:53,764::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:17:53,766::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:17:53,767::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:17:53,768::task::1172::TaskManager.Task::(prepare) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:17:53,768::task::588::TaskManager.Task::(_updateState) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:17:53,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:17:53,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:17:53,769::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:17:53,769::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:17:53,769::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:17:53,769::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:17:53,769::task::978::TaskManager.Task::(_decref) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:17:53,770::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:53,770::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:53,771::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:17:53,771::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:17:55,869::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:17:56,432::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2187::DEBUG::2012-08-30 13:17:57,375::task::588::TaskManager.Task::(_updateState) Task=`09847a8e-5db9-4459-82fd-0d9d0bc9fb9e`::moving from state init -> state preparing >Thread-2187::INFO::2012-08-30 13:17:57,375::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2187::INFO::2012-08-30 13:17:57,376::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178003311157', 'lastCheck': 1346303871.331373, 'code': 0, 'valid': True}} >Thread-2187::DEBUG::2012-08-30 13:17:57,376::task::1172::TaskManager.Task::(prepare) Task=`09847a8e-5db9-4459-82fd-0d9d0bc9fb9e`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178003311157', 'lastCheck': 1346303871.331373, 'code': 0, 'valid': True}} >Thread-2187::DEBUG::2012-08-30 13:17:57,376::task::588::TaskManager.Task::(_updateState) Task=`09847a8e-5db9-4459-82fd-0d9d0bc9fb9e`::moving from state preparing -> state finished >Thread-2187::DEBUG::2012-08-30 13:17:57,376::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2187::DEBUG::2012-08-30 13:17:57,377::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2187::DEBUG::2012-08-30 13:17:57,377::task::978::TaskManager.Task::(_decref) Task=`09847a8e-5db9-4459-82fd-0d9d0bc9fb9e`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:17:58,772::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2188::DEBUG::2012-08-30 13:18:00,333::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2188::DEBUG::2012-08-30 13:18:00,333::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:18:00,870::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:00,870::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2189::DEBUG::2012-08-30 13:18:01,904::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2189::DEBUG::2012-08-30 13:18:01,904::task::588::TaskManager.Task::(_updateState) Task=`8394b7fb-ffeb-4f3c-b994-5a59d5e29daf`::moving from state init -> state preparing >Thread-2189::INFO::2012-08-30 13:18:01,904::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2189::INFO::2012-08-30 13:18:01,905::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2189::DEBUG::2012-08-30 13:18:01,905::task::1172::TaskManager.Task::(prepare) Task=`8394b7fb-ffeb-4f3c-b994-5a59d5e29daf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2189::DEBUG::2012-08-30 13:18:01,905::task::588::TaskManager.Task::(_updateState) Task=`8394b7fb-ffeb-4f3c-b994-5a59d5e29daf`::moving from state preparing -> state finished >Thread-2189::DEBUG::2012-08-30 13:18:01,905::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2189::DEBUG::2012-08-30 13:18:01,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2189::DEBUG::2012-08-30 13:18:01,906::task::978::TaskManager.Task::(_decref) Task=`8394b7fb-ffeb-4f3c-b994-5a59d5e29daf`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:18:03,439::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:18:03,774::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2190::DEBUG::2012-08-30 13:18:05,140::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2190::DEBUG::2012-08-30 13:18:05,141::task::588::TaskManager.Task::(_updateState) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::moving from state init -> state preparing >Thread-2190::INFO::2012-08-30 13:18:05,142::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2190::DEBUG::2012-08-30 13:18:05,142::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`314ae09b-3a07-404e-9f3a-9c3f4a28cdb5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2190::DEBUG::2012-08-30 13:18:05,142::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2190::DEBUG::2012-08-30 13:18:05,142::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2190::DEBUG::2012-08-30 13:18:05,143::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`314ae09b-3a07-404e-9f3a-9c3f4a28cdb5`::Granted request >Thread-2190::DEBUG::2012-08-30 13:18:05,143::task::817::TaskManager.Task::(resourceAcquired) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2190::DEBUG::2012-08-30 13:18:05,143::task::978::TaskManager.Task::(_decref) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::ref 1 aborting False >Thread-2190::INFO::2012-08-30 13:18:05,146::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2190::DEBUG::2012-08-30 13:18:05,146::task::1172::TaskManager.Task::(prepare) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2190::DEBUG::2012-08-30 13:18:05,147::task::588::TaskManager.Task::(_updateState) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::moving from state preparing -> state finished >Thread-2190::DEBUG::2012-08-30 13:18:05,147::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2190::DEBUG::2012-08-30 13:18:05,147::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2190::DEBUG::2012-08-30 13:18:05,147::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2190::DEBUG::2012-08-30 13:18:05,148::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2190::DEBUG::2012-08-30 13:18:05,148::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2190::DEBUG::2012-08-30 13:18:05,148::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2190::DEBUG::2012-08-30 13:18:05,148::task::978::TaskManager.Task::(_decref) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:18:05,871::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:08,775::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:08,775::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:10,872::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:13,776::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:15,873::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:15,874::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2194::DEBUG::2012-08-30 13:18:18,330::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2194::DEBUG::2012-08-30 13:18:18,331::task::588::TaskManager.Task::(_updateState) Task=`6c3e817a-6470-4710-87b7-196d21f0b80a`::moving from state init -> state preparing >Thread-2194::INFO::2012-08-30 13:18:18,331::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2194::INFO::2012-08-30 13:18:18,331::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2194::DEBUG::2012-08-30 13:18:18,332::task::1172::TaskManager.Task::(prepare) Task=`6c3e817a-6470-4710-87b7-196d21f0b80a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2194::DEBUG::2012-08-30 13:18:18,332::task::588::TaskManager.Task::(_updateState) Task=`6c3e817a-6470-4710-87b7-196d21f0b80a`::moving from state preparing -> state finished >Thread-2194::DEBUG::2012-08-30 13:18:18,332::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2194::DEBUG::2012-08-30 13:18:18,332::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2194::DEBUG::2012-08-30 13:18:18,333::task::978::TaskManager.Task::(_decref) Task=`6c3e817a-6470-4710-87b7-196d21f0b80a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:18:18,778::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:20,874::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2196::DEBUG::2012-08-30 13:18:21,582::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2196::DEBUG::2012-08-30 13:18:21,582::task::588::TaskManager.Task::(_updateState) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::moving from state init -> state preparing >Thread-2196::INFO::2012-08-30 13:18:21,583::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2196::DEBUG::2012-08-30 13:18:21,583::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0255acb-9f7a-4ae4-b5ef-51ac8c71899c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2196::DEBUG::2012-08-30 13:18:21,583::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2196::DEBUG::2012-08-30 13:18:21,583::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2196::DEBUG::2012-08-30 13:18:21,584::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0255acb-9f7a-4ae4-b5ef-51ac8c71899c`::Granted request >Thread-2196::DEBUG::2012-08-30 13:18:21,584::task::817::TaskManager.Task::(resourceAcquired) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2196::DEBUG::2012-08-30 13:18:21,584::task::978::TaskManager.Task::(_decref) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::ref 1 aborting False >Thread-2196::INFO::2012-08-30 13:18:21,587::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2196::DEBUG::2012-08-30 13:18:21,587::task::1172::TaskManager.Task::(prepare) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2196::DEBUG::2012-08-30 13:18:21,588::task::588::TaskManager.Task::(_updateState) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::moving from state preparing -> state finished >Thread-2196::DEBUG::2012-08-30 13:18:21,588::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2196::DEBUG::2012-08-30 13:18:21,588::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2196::DEBUG::2012-08-30 13:18:21,588::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2196::DEBUG::2012-08-30 13:18:21,589::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2196::DEBUG::2012-08-30 13:18:21,589::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2196::DEBUG::2012-08-30 13:18:21,589::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2196::DEBUG::2012-08-30 13:18:21,589::task::978::TaskManager.Task::(_decref) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:18:23,779::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:23,780::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:25,875::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2197::DEBUG::2012-08-30 13:18:26,043::task::588::TaskManager.Task::(_updateState) Task=`a426d25a-7a15-40c8-ab77-fc251d0cf268`::moving from state init -> state preparing >Thread-2197::INFO::2012-08-30 13:18:26,044::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2197::INFO::2012-08-30 13:18:26,044::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346303901.3513861, 'code': 0, 'valid': True}} >Thread-2197::DEBUG::2012-08-30 13:18:26,044::task::1172::TaskManager.Task::(prepare) Task=`a426d25a-7a15-40c8-ab77-fc251d0cf268`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346303901.3513861, 'code': 0, 'valid': True}} >Thread-2197::DEBUG::2012-08-30 13:18:26,044::task::588::TaskManager.Task::(_updateState) Task=`a426d25a-7a15-40c8-ab77-fc251d0cf268`::moving from state preparing -> state finished >Thread-2197::DEBUG::2012-08-30 13:18:26,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2197::DEBUG::2012-08-30 13:18:26,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2197::DEBUG::2012-08-30 13:18:26,045::task::978::TaskManager.Task::(_decref) Task=`a426d25a-7a15-40c8-ab77-fc251d0cf268`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:18:26,459::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:18:28,781::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2198::DEBUG::2012-08-30 13:18:29,192::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2198::DEBUG::2012-08-30 13:18:29,193::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::DEBUG::2012-08-30 13:18:30,876::task::588::TaskManager.Task::(_updateState) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:18:30,877::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:18:30,877::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`53ae24c0-0b3f-44bd-9387-f9b0f56acd48`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:18:30,877::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:18:30,878::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:18:30,878::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`53ae24c0-0b3f-44bd-9387-f9b0f56acd48`::Granted request >Thread-104::DEBUG::2012-08-30 13:18:30,878::task::817::TaskManager.Task::(resourceAcquired) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:18:30,879::task::978::TaskManager.Task::(_decref) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:18:30,880::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:18:30,882::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:18:30,883::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:18:30,883::task::1172::TaskManager.Task::(prepare) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:18:30,884::task::588::TaskManager.Task::(_updateState) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:18:30,884::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:18:30,884::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:18:30,884::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:18:30,885::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:18:30,885::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:18:30,885::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:18:30,885::task::978::TaskManager.Task::(_decref) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:18:30,886::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:30,886::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:30,887::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:30,887::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:18:33,465::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:18:33,782::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2200::DEBUG::2012-08-30 13:18:34,442::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2200::DEBUG::2012-08-30 13:18:34,442::task::588::TaskManager.Task::(_updateState) Task=`9481aa17-b705-43f3-9a5d-388d693f327a`::moving from state init -> state preparing >Thread-2200::INFO::2012-08-30 13:18:34,443::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2200::INFO::2012-08-30 13:18:34,443::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2200::DEBUG::2012-08-30 13:18:34,443::task::1172::TaskManager.Task::(prepare) Task=`9481aa17-b705-43f3-9a5d-388d693f327a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2200::DEBUG::2012-08-30 13:18:34,443::task::588::TaskManager.Task::(_updateState) Task=`9481aa17-b705-43f3-9a5d-388d693f327a`::moving from state preparing -> state finished >Thread-2200::DEBUG::2012-08-30 13:18:34,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2200::DEBUG::2012-08-30 13:18:34,444::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2200::DEBUG::2012-08-30 13:18:34,444::task::978::TaskManager.Task::(_decref) Task=`9481aa17-b705-43f3-9a5d-388d693f327a`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:18:35,888::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2201::DEBUG::2012-08-30 13:18:37,616::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2201::DEBUG::2012-08-30 13:18:37,617::task::588::TaskManager.Task::(_updateState) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::moving from state init -> state preparing >Thread-2201::INFO::2012-08-30 13:18:37,617::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2201::DEBUG::2012-08-30 13:18:37,617::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`10004e12-6aa8-4bac-a91d-3140fa3fdffe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2201::DEBUG::2012-08-30 13:18:37,617::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2201::DEBUG::2012-08-30 13:18:37,618::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2201::DEBUG::2012-08-30 13:18:37,618::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`10004e12-6aa8-4bac-a91d-3140fa3fdffe`::Granted request >Thread-2201::DEBUG::2012-08-30 13:18:37,619::task::817::TaskManager.Task::(resourceAcquired) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2201::DEBUG::2012-08-30 13:18:37,619::task::978::TaskManager.Task::(_decref) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::ref 1 aborting False >Thread-2201::INFO::2012-08-30 13:18:37,622::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2201::DEBUG::2012-08-30 13:18:37,622::task::1172::TaskManager.Task::(prepare) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2201::DEBUG::2012-08-30 13:18:37,622::task::588::TaskManager.Task::(_updateState) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::moving from state preparing -> state finished >Thread-2201::DEBUG::2012-08-30 13:18:37,623::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2201::DEBUG::2012-08-30 13:18:37,623::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2201::DEBUG::2012-08-30 13:18:37,623::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2201::DEBUG::2012-08-30 13:18:37,623::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2201::DEBUG::2012-08-30 13:18:37,624::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2201::DEBUG::2012-08-30 13:18:37,624::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2201::DEBUG::2012-08-30 13:18:37,624::task::978::TaskManager.Task::(_decref) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:18:38,783::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:38,784::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:40,889::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:43,785::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:45,890::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:18:45,890::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:48,786::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2205::DEBUG::2012-08-30 13:18:50,807::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2205::DEBUG::2012-08-30 13:18:50,807::task::588::TaskManager.Task::(_updateState) Task=`afa4ea4a-0968-4d0a-97df-4bd41540d00e`::moving from state init -> state preparing >Thread-2205::INFO::2012-08-30 13:18:50,808::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2205::INFO::2012-08-30 13:18:50,808::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2205::DEBUG::2012-08-30 13:18:50,808::task::1172::TaskManager.Task::(prepare) Task=`afa4ea4a-0968-4d0a-97df-4bd41540d00e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2205::DEBUG::2012-08-30 13:18:50,808::task::588::TaskManager.Task::(_updateState) Task=`afa4ea4a-0968-4d0a-97df-4bd41540d00e`::moving from state preparing -> state finished >Thread-2205::DEBUG::2012-08-30 13:18:50,809::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2205::DEBUG::2012-08-30 13:18:50,809::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2205::DEBUG::2012-08-30 13:18:50,809::task::978::TaskManager.Task::(_decref) Task=`afa4ea4a-0968-4d0a-97df-4bd41540d00e`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:18:50,891::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:18:53,787::task::588::TaskManager.Task::(_updateState) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:18:53,788::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:18:53,789::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`2b0735cc-e720-41b9-9499-8c303f6bca00`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:18:53,789::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:18:53,789::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:18:53,789::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`2b0735cc-e720-41b9-9499-8c303f6bca00`::Granted request >Thread-40::DEBUG::2012-08-30 13:18:53,790::task::817::TaskManager.Task::(resourceAcquired) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:18:53,790::task::978::TaskManager.Task::(_decref) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:18:53,792::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:18:53,794::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:18:53,795::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:18:53,795::task::1172::TaskManager.Task::(prepare) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:18:53,795::task::588::TaskManager.Task::(_updateState) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:18:53,797::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:18:53,797::task::978::TaskManager.Task::(_decref) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:18:53,797::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:53,798::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:53,798::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:18:53,799::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2206::DEBUG::2012-08-30 13:18:54,215::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2207::DEBUG::2012-08-30 13:18:54,216::task::588::TaskManager.Task::(_updateState) Task=`c0a132de-5024-4f4e-9d61-3a0e7506d67c`::moving from state init -> state preparing >Thread-2206::DEBUG::2012-08-30 13:18:54,216::task::588::TaskManager.Task::(_updateState) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::moving from state init -> state preparing >Thread-2207::INFO::2012-08-30 13:18:54,216::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2206::INFO::2012-08-30 13:18:54,216::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2207::INFO::2012-08-30 13:18:54,217::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176405906677', 'lastCheck': 1346303931.3708031, 'code': 0, 'valid': True}} >Thread-2206::DEBUG::2012-08-30 13:18:54,217::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0a7a9b3d-1544-415c-8e60-5ff616390283`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2207::DEBUG::2012-08-30 13:18:54,217::task::1172::TaskManager.Task::(prepare) Task=`c0a132de-5024-4f4e-9d61-3a0e7506d67c`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176405906677', 'lastCheck': 1346303931.3708031, 'code': 0, 'valid': True}} >Thread-2206::DEBUG::2012-08-30 13:18:54,217::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2207::DEBUG::2012-08-30 13:18:54,218::task::588::TaskManager.Task::(_updateState) Task=`c0a132de-5024-4f4e-9d61-3a0e7506d67c`::moving from state preparing -> state finished >Thread-2206::DEBUG::2012-08-30 13:18:54,218::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2207::DEBUG::2012-08-30 13:18:54,218::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2206::DEBUG::2012-08-30 13:18:54,218::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0a7a9b3d-1544-415c-8e60-5ff616390283`::Granted request >Thread-2207::DEBUG::2012-08-30 13:18:54,218::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2206::DEBUG::2012-08-30 13:18:54,219::task::817::TaskManager.Task::(resourceAcquired) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2207::DEBUG::2012-08-30 13:18:54,219::task::978::TaskManager.Task::(_decref) Task=`c0a132de-5024-4f4e-9d61-3a0e7506d67c`::ref 0 aborting False >Thread-2206::DEBUG::2012-08-30 13:18:54,219::task::978::TaskManager.Task::(_decref) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::ref 1 aborting False >Thread-2206::INFO::2012-08-30 13:18:54,224::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2206::DEBUG::2012-08-30 13:18:54,224::task::1172::TaskManager.Task::(prepare) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2206::DEBUG::2012-08-30 13:18:54,224::task::588::TaskManager.Task::(_updateState) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::moving from state preparing -> state finished >Thread-2206::DEBUG::2012-08-30 13:18:54,225::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2206::DEBUG::2012-08-30 13:18:54,225::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2206::DEBUG::2012-08-30 13:18:54,225::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2206::DEBUG::2012-08-30 13:18:54,225::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2206::DEBUG::2012-08-30 13:18:54,226::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2206::DEBUG::2012-08-30 13:18:54,226::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2206::DEBUG::2012-08-30 13:18:54,226::task::978::TaskManager.Task::(_decref) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:18:55,892::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:18:56,486::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2208::DEBUG::2012-08-30 13:18:57,387::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2208::DEBUG::2012-08-30 13:18:57,387::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:18:58,800::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:00,893::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:00,894::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:19:03,493::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:19:03,801::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:05,895::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2210::DEBUG::2012-08-30 13:19:07,446::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2210::DEBUG::2012-08-30 13:19:07,446::task::588::TaskManager.Task::(_updateState) Task=`efe4b058-52f8-4045-bff1-4c28660c5ae7`::moving from state init -> state preparing >Thread-2210::INFO::2012-08-30 13:19:07,447::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2210::INFO::2012-08-30 13:19:07,447::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2210::DEBUG::2012-08-30 13:19:07,447::task::1172::TaskManager.Task::(prepare) Task=`efe4b058-52f8-4045-bff1-4c28660c5ae7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2210::DEBUG::2012-08-30 13:19:07,447::task::588::TaskManager.Task::(_updateState) Task=`efe4b058-52f8-4045-bff1-4c28660c5ae7`::moving from state preparing -> state finished >Thread-2210::DEBUG::2012-08-30 13:19:07,448::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2210::DEBUG::2012-08-30 13:19:07,448::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2210::DEBUG::2012-08-30 13:19:07,448::task::978::TaskManager.Task::(_decref) Task=`efe4b058-52f8-4045-bff1-4c28660c5ae7`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:19:08,802::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:08,803::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2212::DEBUG::2012-08-30 13:19:10,636::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2212::DEBUG::2012-08-30 13:19:10,636::task::588::TaskManager.Task::(_updateState) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::moving from state init -> state preparing >Thread-2212::INFO::2012-08-30 13:19:10,636::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2212::DEBUG::2012-08-30 13:19:10,637::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e61e5291-17eb-4fb8-a881-fbae1d660ee6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2212::DEBUG::2012-08-30 13:19:10,637::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2212::DEBUG::2012-08-30 13:19:10,637::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2212::DEBUG::2012-08-30 13:19:10,637::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e61e5291-17eb-4fb8-a881-fbae1d660ee6`::Granted request >Thread-2212::DEBUG::2012-08-30 13:19:10,638::task::817::TaskManager.Task::(resourceAcquired) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2212::DEBUG::2012-08-30 13:19:10,638::task::978::TaskManager.Task::(_decref) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::ref 1 aborting False >Thread-2212::INFO::2012-08-30 13:19:10,641::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2212::DEBUG::2012-08-30 13:19:10,641::task::1172::TaskManager.Task::(prepare) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2212::DEBUG::2012-08-30 13:19:10,641::task::588::TaskManager.Task::(_updateState) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::moving from state preparing -> state finished >Thread-2212::DEBUG::2012-08-30 13:19:10,642::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2212::DEBUG::2012-08-30 13:19:10,642::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2212::DEBUG::2012-08-30 13:19:10,642::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2212::DEBUG::2012-08-30 13:19:10,642::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2212::DEBUG::2012-08-30 13:19:10,643::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2212::DEBUG::2012-08-30 13:19:10,643::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2212::DEBUG::2012-08-30 13:19:10,643::task::978::TaskManager.Task::(_decref) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:19:10,896::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:13,804::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:15,896::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:15,897::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:18,805::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:20,898::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2215::DEBUG::2012-08-30 13:19:22,410::task::588::TaskManager.Task::(_updateState) Task=`6420cfc3-02ab-47ed-be74-4fc0abdee7a5`::moving from state init -> state preparing >Thread-2215::INFO::2012-08-30 13:19:22,410::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2215::INFO::2012-08-30 13:19:22,410::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303961.3905671, 'code': 0, 'valid': True}} >Thread-2215::DEBUG::2012-08-30 13:19:22,410::task::1172::TaskManager.Task::(prepare) Task=`6420cfc3-02ab-47ed-be74-4fc0abdee7a5`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303961.3905671, 'code': 0, 'valid': True}} >Thread-2215::DEBUG::2012-08-30 13:19:22,411::task::588::TaskManager.Task::(_updateState) Task=`6420cfc3-02ab-47ed-be74-4fc0abdee7a5`::moving from state preparing -> state finished >Thread-2215::DEBUG::2012-08-30 13:19:22,411::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2215::DEBUG::2012-08-30 13:19:22,411::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2215::DEBUG::2012-08-30 13:19:22,411::task::978::TaskManager.Task::(_decref) Task=`6420cfc3-02ab-47ed-be74-4fc0abdee7a5`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:19:23,806::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:23,807::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2216::DEBUG::2012-08-30 13:19:23,970::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2216::DEBUG::2012-08-30 13:19:23,970::task::588::TaskManager.Task::(_updateState) Task=`aa12be73-2634-4bde-aeac-2e179fc9c524`::moving from state init -> state preparing >Thread-2216::INFO::2012-08-30 13:19:23,971::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2216::INFO::2012-08-30 13:19:23,971::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2216::DEBUG::2012-08-30 13:19:23,971::task::1172::TaskManager.Task::(prepare) Task=`aa12be73-2634-4bde-aeac-2e179fc9c524`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2216::DEBUG::2012-08-30 13:19:23,971::task::588::TaskManager.Task::(_updateState) Task=`aa12be73-2634-4bde-aeac-2e179fc9c524`::moving from state preparing -> state finished >Thread-2216::DEBUG::2012-08-30 13:19:23,972::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2216::DEBUG::2012-08-30 13:19:23,972::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2216::DEBUG::2012-08-30 13:19:23,972::task::978::TaskManager.Task::(_decref) Task=`aa12be73-2634-4bde-aeac-2e179fc9c524`::ref 0 aborting False >Thread-2217::DEBUG::2012-08-30 13:19:25,539::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2217::DEBUG::2012-08-30 13:19:25,539::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:19:25,899::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:19:26,514::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2218::DEBUG::2012-08-30 13:19:27,147::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2218::DEBUG::2012-08-30 13:19:27,148::task::588::TaskManager.Task::(_updateState) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::moving from state init -> state preparing >Thread-2218::INFO::2012-08-30 13:19:27,148::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2218::DEBUG::2012-08-30 13:19:27,148::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5d4603fb-8ae1-4822-9ece-992f1287559b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2218::DEBUG::2012-08-30 13:19:27,148::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2218::DEBUG::2012-08-30 13:19:27,149::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2218::DEBUG::2012-08-30 13:19:27,149::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5d4603fb-8ae1-4822-9ece-992f1287559b`::Granted request >Thread-2218::DEBUG::2012-08-30 13:19:27,149::task::817::TaskManager.Task::(resourceAcquired) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2218::DEBUG::2012-08-30 13:19:27,149::task::978::TaskManager.Task::(_decref) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::ref 1 aborting False >Thread-2218::INFO::2012-08-30 13:19:27,152::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2218::DEBUG::2012-08-30 13:19:27,153::task::1172::TaskManager.Task::(prepare) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2218::DEBUG::2012-08-30 13:19:27,153::task::588::TaskManager.Task::(_updateState) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::moving from state preparing -> state finished >Thread-2218::DEBUG::2012-08-30 13:19:27,153::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2218::DEBUG::2012-08-30 13:19:27,153::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2218::DEBUG::2012-08-30 13:19:27,154::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2218::DEBUG::2012-08-30 13:19:27,154::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2218::DEBUG::2012-08-30 13:19:27,154::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2218::DEBUG::2012-08-30 13:19:27,154::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2218::DEBUG::2012-08-30 13:19:27,155::task::978::TaskManager.Task::(_decref) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:19:28,808::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:19:30,900::task::588::TaskManager.Task::(_updateState) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:19:30,900::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:19:30,900::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`dbc9cdb3-9e41-45fd-b44f-8de6b35d0ace`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:19:30,901::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:19:30,901::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:19:30,901::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`dbc9cdb3-9e41-45fd-b44f-8de6b35d0ace`::Granted request >Thread-104::DEBUG::2012-08-30 13:19:30,901::task::817::TaskManager.Task::(resourceAcquired) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:19:30,902::task::978::TaskManager.Task::(_decref) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:19:30,903::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:19:30,905::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:19:30,906::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:19:30,907::task::1172::TaskManager.Task::(prepare) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:19:30,907::task::588::TaskManager.Task::(_updateState) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:19:30,907::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:19:30,907::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:19:30,908::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:19:30,908::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:19:30,908::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:19:30,908::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:19:30,909::task::978::TaskManager.Task::(_decref) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:19:30,909::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:30,909::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:30,910::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:30,910::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:19:33,522::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:19:33,809::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:35,911::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:38,810::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:38,811::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2221::DEBUG::2012-08-30 13:19:40,425::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2221::DEBUG::2012-08-30 13:19:40,425::task::588::TaskManager.Task::(_updateState) Task=`3a91d453-2933-4caa-9f2f-811c1592aa1c`::moving from state init -> state preparing >Thread-2221::INFO::2012-08-30 13:19:40,425::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2221::INFO::2012-08-30 13:19:40,426::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2221::DEBUG::2012-08-30 13:19:40,426::task::1172::TaskManager.Task::(prepare) Task=`3a91d453-2933-4caa-9f2f-811c1592aa1c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2221::DEBUG::2012-08-30 13:19:40,426::task::588::TaskManager.Task::(_updateState) Task=`3a91d453-2933-4caa-9f2f-811c1592aa1c`::moving from state preparing -> state finished >Thread-2221::DEBUG::2012-08-30 13:19:40,426::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2221::DEBUG::2012-08-30 13:19:40,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2221::DEBUG::2012-08-30 13:19:40,427::task::978::TaskManager.Task::(_decref) Task=`3a91d453-2933-4caa-9f2f-811c1592aa1c`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:19:40,912::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2223::DEBUG::2012-08-30 13:19:43,585::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2223::DEBUG::2012-08-30 13:19:43,585::task::588::TaskManager.Task::(_updateState) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::moving from state init -> state preparing >Thread-2223::INFO::2012-08-30 13:19:43,585::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2223::DEBUG::2012-08-30 13:19:43,586::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9aa7717c-9ff8-4353-b45f-2e0d21cd9a7b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2223::DEBUG::2012-08-30 13:19:43,586::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2223::DEBUG::2012-08-30 13:19:43,586::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2223::DEBUG::2012-08-30 13:19:43,586::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9aa7717c-9ff8-4353-b45f-2e0d21cd9a7b`::Granted request >Thread-2223::DEBUG::2012-08-30 13:19:43,587::task::817::TaskManager.Task::(resourceAcquired) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2223::DEBUG::2012-08-30 13:19:43,587::task::978::TaskManager.Task::(_decref) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::ref 1 aborting False >Thread-2223::INFO::2012-08-30 13:19:43,590::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2223::DEBUG::2012-08-30 13:19:43,590::task::1172::TaskManager.Task::(prepare) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2223::DEBUG::2012-08-30 13:19:43,590::task::588::TaskManager.Task::(_updateState) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::moving from state preparing -> state finished >Thread-2223::DEBUG::2012-08-30 13:19:43,591::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2223::DEBUG::2012-08-30 13:19:43,591::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2223::DEBUG::2012-08-30 13:19:43,591::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2223::DEBUG::2012-08-30 13:19:43,591::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2223::DEBUG::2012-08-30 13:19:43,592::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2223::DEBUG::2012-08-30 13:19:43,592::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2223::DEBUG::2012-08-30 13:19:43,592::task::978::TaskManager.Task::(_decref) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:19:43,812::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:45,913::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:45,913::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:48,813::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:19:50,914::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2225::DEBUG::2012-08-30 13:19:51,503::task::588::TaskManager.Task::(_updateState) Task=`23c75423-035c-4585-8987-dd0cdae398cb`::moving from state init -> state preparing >Thread-2225::INFO::2012-08-30 13:19:51,503::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2225::INFO::2012-08-30 13:19:51,503::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176191329956', 'lastCheck': 1346303991.4098279, 'code': 0, 'valid': True}} >Thread-2225::DEBUG::2012-08-30 13:19:51,503::task::1172::TaskManager.Task::(prepare) Task=`23c75423-035c-4585-8987-dd0cdae398cb`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176191329956', 'lastCheck': 1346303991.4098279, 'code': 0, 'valid': True}} >Thread-2225::DEBUG::2012-08-30 13:19:51,504::task::588::TaskManager.Task::(_updateState) Task=`23c75423-035c-4585-8987-dd0cdae398cb`::moving from state preparing -> state finished >Thread-2225::DEBUG::2012-08-30 13:19:51,504::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2225::DEBUG::2012-08-30 13:19:51,504::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2225::DEBUG::2012-08-30 13:19:51,504::task::978::TaskManager.Task::(_decref) Task=`23c75423-035c-4585-8987-dd0cdae398cb`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 13:19:53,814::task::588::TaskManager.Task::(_updateState) Task=`03e9a479-1232-442f-9b24-4653460e0153`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:19:53,815::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:19:53,815::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4ed66c77-f428-4c03-a363-4fd036a1d25a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:19:53,815::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:19:53,816::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:19:53,816::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4ed66c77-f428-4c03-a363-4fd036a1d25a`::Granted request >Thread-40::DEBUG::2012-08-30 13:19:53,816::task::817::TaskManager.Task::(resourceAcquired) Task=`03e9a479-1232-442f-9b24-4653460e0153`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:19:53,816::task::978::TaskManager.Task::(_decref) Task=`03e9a479-1232-442f-9b24-4653460e0153`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:19:53,818::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:19:53,820::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:19:53,821::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:19:53,822::task::1172::TaskManager.Task::(prepare) Task=`03e9a479-1232-442f-9b24-4653460e0153`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:19:53,822::task::588::TaskManager.Task::(_updateState) Task=`03e9a479-1232-442f-9b24-4653460e0153`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:19:53,822::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:19:53,822::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:19:53,823::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:19:53,823::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:19:53,823::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:19:53,823::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:19:53,824::task::978::TaskManager.Task::(_decref) Task=`03e9a479-1232-442f-9b24-4653460e0153`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:19:53,824::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:53,824::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:53,825::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:19:53,825::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2226::DEBUG::2012-08-30 13:19:54,683::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2226::DEBUG::2012-08-30 13:19:54,683::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:19:55,915::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2227::DEBUG::2012-08-30 13:19:56,253::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2227::DEBUG::2012-08-30 13:19:56,254::task::588::TaskManager.Task::(_updateState) Task=`832c4dc4-1c66-41bf-bdd6-4872327d4dda`::moving from state init -> state preparing >Thread-2227::INFO::2012-08-30 13:19:56,254::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2227::INFO::2012-08-30 13:19:56,254::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2227::DEBUG::2012-08-30 13:19:56,254::task::1172::TaskManager.Task::(prepare) Task=`832c4dc4-1c66-41bf-bdd6-4872327d4dda`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2227::DEBUG::2012-08-30 13:19:56,255::task::588::TaskManager.Task::(_updateState) Task=`832c4dc4-1c66-41bf-bdd6-4872327d4dda`::moving from state preparing -> state finished >Thread-2227::DEBUG::2012-08-30 13:19:56,255::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2227::DEBUG::2012-08-30 13:19:56,255::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2227::DEBUG::2012-08-30 13:19:56,255::task::978::TaskManager.Task::(_decref) Task=`832c4dc4-1c66-41bf-bdd6-4872327d4dda`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:19:56,543::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:19:58,826::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2228::DEBUG::2012-08-30 13:19:59,423::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2228::DEBUG::2012-08-30 13:19:59,423::task::588::TaskManager.Task::(_updateState) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::moving from state init -> state preparing >Thread-2228::INFO::2012-08-30 13:19:59,424::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2228::DEBUG::2012-08-30 13:19:59,424::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`22e1306f-997e-4b56-b68b-e3466cf4d337`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2228::DEBUG::2012-08-30 13:19:59,425::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2228::DEBUG::2012-08-30 13:19:59,425::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2228::DEBUG::2012-08-30 13:19:59,425::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`22e1306f-997e-4b56-b68b-e3466cf4d337`::Granted request >Thread-2228::DEBUG::2012-08-30 13:19:59,426::task::817::TaskManager.Task::(resourceAcquired) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2228::DEBUG::2012-08-30 13:19:59,426::task::978::TaskManager.Task::(_decref) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::ref 1 aborting False >Thread-2228::INFO::2012-08-30 13:19:59,429::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2228::DEBUG::2012-08-30 13:19:59,429::task::1172::TaskManager.Task::(prepare) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2228::DEBUG::2012-08-30 13:19:59,429::task::588::TaskManager.Task::(_updateState) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::moving from state preparing -> state finished >Thread-2228::DEBUG::2012-08-30 13:19:59,429::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2228::DEBUG::2012-08-30 13:19:59,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2228::DEBUG::2012-08-30 13:19:59,430::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2228::DEBUG::2012-08-30 13:19:59,430::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2228::DEBUG::2012-08-30 13:19:59,430::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2228::DEBUG::2012-08-30 13:19:59,431::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2228::DEBUG::2012-08-30 13:19:59,431::task::978::TaskManager.Task::(_decref) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:20:00,916::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:00,917::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:20:03,550::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:20:03,827::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:05,918::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:08,828::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:08,829::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:10,919::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:13,830::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2232::DEBUG::2012-08-30 13:20:14,656::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2232::DEBUG::2012-08-30 13:20:14,657::task::588::TaskManager.Task::(_updateState) Task=`434d9542-c7e0-4488-860a-93cc272b376f`::moving from state init -> state preparing >Thread-2232::INFO::2012-08-30 13:20:14,657::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2232::INFO::2012-08-30 13:20:14,657::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2232::DEBUG::2012-08-30 13:20:14,657::task::1172::TaskManager.Task::(prepare) Task=`434d9542-c7e0-4488-860a-93cc272b376f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2232::DEBUG::2012-08-30 13:20:14,658::task::588::TaskManager.Task::(_updateState) Task=`434d9542-c7e0-4488-860a-93cc272b376f`::moving from state preparing -> state finished >Thread-2232::DEBUG::2012-08-30 13:20:14,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2232::DEBUG::2012-08-30 13:20:14,658::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2232::DEBUG::2012-08-30 13:20:14,658::task::978::TaskManager.Task::(_decref) Task=`434d9542-c7e0-4488-860a-93cc272b376f`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:20:15,920::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:15,920::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2234::DEBUG::2012-08-30 13:20:17,819::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2234::DEBUG::2012-08-30 13:20:17,819::task::588::TaskManager.Task::(_updateState) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::moving from state init -> state preparing >Thread-2234::INFO::2012-08-30 13:20:17,820::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2234::DEBUG::2012-08-30 13:20:17,820::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ad7c4be3-32e1-43cc-86d6-90d0f8da26c0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2234::DEBUG::2012-08-30 13:20:17,820::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2234::DEBUG::2012-08-30 13:20:17,820::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2234::DEBUG::2012-08-30 13:20:17,821::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ad7c4be3-32e1-43cc-86d6-90d0f8da26c0`::Granted request >Thread-2234::DEBUG::2012-08-30 13:20:17,821::task::817::TaskManager.Task::(resourceAcquired) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2234::DEBUG::2012-08-30 13:20:17,821::task::978::TaskManager.Task::(_decref) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::ref 1 aborting False >Thread-2234::INFO::2012-08-30 13:20:17,824::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2234::DEBUG::2012-08-30 13:20:17,824::task::1172::TaskManager.Task::(prepare) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2234::DEBUG::2012-08-30 13:20:17,825::task::588::TaskManager.Task::(_updateState) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::moving from state preparing -> state finished >Thread-2234::DEBUG::2012-08-30 13:20:17,825::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2234::DEBUG::2012-08-30 13:20:17,825::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2234::DEBUG::2012-08-30 13:20:17,825::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2234::DEBUG::2012-08-30 13:20:17,826::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2234::DEBUG::2012-08-30 13:20:17,826::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2234::DEBUG::2012-08-30 13:20:17,826::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2234::DEBUG::2012-08-30 13:20:17,826::task::978::TaskManager.Task::(_decref) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:20:18,831::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2235::DEBUG::2012-08-30 13:20:19,405::task::588::TaskManager.Task::(_updateState) Task=`20ef9148-7bdf-478b-a1e9-c6990cb7d482`::moving from state init -> state preparing >Thread-2235::INFO::2012-08-30 13:20:19,405::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2235::INFO::2012-08-30 13:20:19,406::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346304011.423502, 'code': 0, 'valid': True}} >Thread-2235::DEBUG::2012-08-30 13:20:19,406::task::1172::TaskManager.Task::(prepare) Task=`20ef9148-7bdf-478b-a1e9-c6990cb7d482`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346304011.423502, 'code': 0, 'valid': True}} >Thread-2235::DEBUG::2012-08-30 13:20:19,406::task::588::TaskManager.Task::(_updateState) Task=`20ef9148-7bdf-478b-a1e9-c6990cb7d482`::moving from state preparing -> state finished >Thread-2235::DEBUG::2012-08-30 13:20:19,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2235::DEBUG::2012-08-30 13:20:19,407::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2235::DEBUG::2012-08-30 13:20:19,407::task::978::TaskManager.Task::(_decref) Task=`20ef9148-7bdf-478b-a1e9-c6990cb7d482`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:20:20,921::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2236::DEBUG::2012-08-30 13:20:22,605::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2236::DEBUG::2012-08-30 13:20:22,605::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:20:23,833::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:23,833::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:25,922::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:20:26,571::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:20:28,835::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:20:30,923::task::588::TaskManager.Task::(_updateState) Task=`def63159-06e2-437d-831f-fe244bdca503`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:20:30,923::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:20:30,923::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`cab92603-a281-43a6-9e8a-27b922366e1c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:20:30,924::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:20:30,924::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:20:30,924::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`cab92603-a281-43a6-9e8a-27b922366e1c`::Granted request >Thread-104::DEBUG::2012-08-30 13:20:30,925::task::817::TaskManager.Task::(resourceAcquired) Task=`def63159-06e2-437d-831f-fe244bdca503`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:20:30,925::task::978::TaskManager.Task::(_decref) Task=`def63159-06e2-437d-831f-fe244bdca503`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:20:30,926::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:20:30,928::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:20:30,930::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:20:30,930::task::1172::TaskManager.Task::(prepare) Task=`def63159-06e2-437d-831f-fe244bdca503`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:20:30,930::task::588::TaskManager.Task::(_updateState) Task=`def63159-06e2-437d-831f-fe244bdca503`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:20:30,930::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:20:30,930::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:20:30,931::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:20:30,931::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:20:30,931::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:20:30,931::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:20:30,932::task::978::TaskManager.Task::(_decref) Task=`def63159-06e2-437d-831f-fe244bdca503`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:20:30,932::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:30,932::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:30,933::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:30,933::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2238::DEBUG::2012-08-30 13:20:31,022::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2238::DEBUG::2012-08-30 13:20:31,023::task::588::TaskManager.Task::(_updateState) Task=`560f5901-3721-4c6d-b6b6-c681b87bf9a7`::moving from state init -> state preparing >Thread-2238::INFO::2012-08-30 13:20:31,023::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2238::INFO::2012-08-30 13:20:31,023::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2238::DEBUG::2012-08-30 13:20:31,023::task::1172::TaskManager.Task::(prepare) Task=`560f5901-3721-4c6d-b6b6-c681b87bf9a7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2238::DEBUG::2012-08-30 13:20:31,024::task::588::TaskManager.Task::(_updateState) Task=`560f5901-3721-4c6d-b6b6-c681b87bf9a7`::moving from state preparing -> state finished >Thread-2238::DEBUG::2012-08-30 13:20:31,024::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2238::DEBUG::2012-08-30 13:20:31,024::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2238::DEBUG::2012-08-30 13:20:31,024::task::978::TaskManager.Task::(_decref) Task=`560f5901-3721-4c6d-b6b6-c681b87bf9a7`::ref 0 aborting False >Thread-26::DEBUG::2012-08-30 13:20:31,429::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:20:31,430::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-26::DEBUG::2012-08-30 13:20:31,430::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None) >Thread-26::DEBUG::2012-08-30 13:20:31,539::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5 >Thread-26::WARNING::2012-08-30 13:20:31,539::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found'] >Thread-26::DEBUG::2012-08-30 13:20:31,540::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-26::DEBUG::2012-08-30 13:20:31,546::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91 >Thread-26::DEBUG::2012-08-30 13:20:31,546::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend >Thread-26::DEBUG::2012-08-30 13:20:31,549::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d'] >Thread-26::DEBUG::2012-08-30 13:20:31,550::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-26::WARNING::2012-08-30 13:20:31,550::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered >Thread-26::WARNING::2012-08-30 13:20:31,550::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered >VM Channels Listener::DEBUG::2012-08-30 13:20:33,578::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:20:33,836::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2240::DEBUG::2012-08-30 13:20:34,327::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2240::DEBUG::2012-08-30 13:20:34,327::task::588::TaskManager.Task::(_updateState) Task=`9da5d875-1de8-4458-ab32-97d215720261`::moving from state init -> state preparing >Thread-2240::INFO::2012-08-30 13:20:34,327::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2240::DEBUG::2012-08-30 13:20:34,328::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f4b03537-fbc8-4a71-9765-3b04b150fb0f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2240::DEBUG::2012-08-30 13:20:34,328::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2240::DEBUG::2012-08-30 13:20:34,328::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2240::DEBUG::2012-08-30 13:20:34,329::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f4b03537-fbc8-4a71-9765-3b04b150fb0f`::Granted request >Thread-2240::DEBUG::2012-08-30 13:20:34,329::task::817::TaskManager.Task::(resourceAcquired) Task=`9da5d875-1de8-4458-ab32-97d215720261`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2240::DEBUG::2012-08-30 13:20:34,329::task::978::TaskManager.Task::(_decref) Task=`9da5d875-1de8-4458-ab32-97d215720261`::ref 1 aborting False >Thread-2240::INFO::2012-08-30 13:20:34,332::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2240::DEBUG::2012-08-30 13:20:34,332::task::1172::TaskManager.Task::(prepare) Task=`9da5d875-1de8-4458-ab32-97d215720261`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2240::DEBUG::2012-08-30 13:20:34,333::task::588::TaskManager.Task::(_updateState) Task=`9da5d875-1de8-4458-ab32-97d215720261`::moving from state preparing -> state finished >Thread-2240::DEBUG::2012-08-30 13:20:34,333::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2240::DEBUG::2012-08-30 13:20:34,333::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2240::DEBUG::2012-08-30 13:20:34,333::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2240::DEBUG::2012-08-30 13:20:34,334::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2240::DEBUG::2012-08-30 13:20:34,334::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2240::DEBUG::2012-08-30 13:20:34,334::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2240::DEBUG::2012-08-30 13:20:34,334::task::978::TaskManager.Task::(_decref) Task=`9da5d875-1de8-4458-ab32-97d215720261`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:20:35,934::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:38,837::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:38,838::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:40,935::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:43,839::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:45,936::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:45,937::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2248::DEBUG::2012-08-30 13:20:47,578::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2249::DEBUG::2012-08-30 13:20:47,578::task::588::TaskManager.Task::(_updateState) Task=`8b500c3f-bc1e-41cc-b2fb-826598deefd3`::moving from state init -> state preparing >Thread-2248::DEBUG::2012-08-30 13:20:47,578::task::588::TaskManager.Task::(_updateState) Task=`e74172cf-89cf-478f-a9b0-8f7fb6d1be9b`::moving from state init -> state preparing >Thread-2249::INFO::2012-08-30 13:20:47,579::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2248::INFO::2012-08-30 13:20:47,579::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2249::INFO::2012-08-30 13:20:47,579::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163316726685', 'lastCheck': 1346304041.563174, 'code': 0, 'valid': True}} >Thread-2248::INFO::2012-08-30 13:20:47,579::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2249::DEBUG::2012-08-30 13:20:47,580::task::1172::TaskManager.Task::(prepare) Task=`8b500c3f-bc1e-41cc-b2fb-826598deefd3`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163316726685', 'lastCheck': 1346304041.563174, 'code': 0, 'valid': True}} >Thread-2248::DEBUG::2012-08-30 13:20:47,580::task::1172::TaskManager.Task::(prepare) Task=`e74172cf-89cf-478f-a9b0-8f7fb6d1be9b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2249::DEBUG::2012-08-30 13:20:47,580::task::588::TaskManager.Task::(_updateState) Task=`8b500c3f-bc1e-41cc-b2fb-826598deefd3`::moving from state preparing -> state finished >Thread-2248::DEBUG::2012-08-30 13:20:47,580::task::588::TaskManager.Task::(_updateState) Task=`e74172cf-89cf-478f-a9b0-8f7fb6d1be9b`::moving from state preparing -> state finished >Thread-2249::DEBUG::2012-08-30 13:20:47,580::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2248::DEBUG::2012-08-30 13:20:47,581::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2249::DEBUG::2012-08-30 13:20:47,581::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2248::DEBUG::2012-08-30 13:20:47,581::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2249::DEBUG::2012-08-30 13:20:47,581::task::978::TaskManager.Task::(_decref) Task=`8b500c3f-bc1e-41cc-b2fb-826598deefd3`::ref 0 aborting False >Thread-2248::DEBUG::2012-08-30 13:20:47,581::task::978::TaskManager.Task::(_decref) Task=`e74172cf-89cf-478f-a9b0-8f7fb6d1be9b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:20:48,841::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2251::DEBUG::2012-08-30 13:20:50,756::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2250::DEBUG::2012-08-30 13:20:50,757::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2251::DEBUG::2012-08-30 13:20:50,757::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-2250::DEBUG::2012-08-30 13:20:50,758::task::588::TaskManager.Task::(_updateState) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::moving from state init -> state preparing >Thread-2250::INFO::2012-08-30 13:20:50,759::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2250::DEBUG::2012-08-30 13:20:50,759::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ef0d8a06-133a-4d5d-a51c-5eec68151cad`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2250::DEBUG::2012-08-30 13:20:50,760::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2250::DEBUG::2012-08-30 13:20:50,760::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2250::DEBUG::2012-08-30 13:20:50,760::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ef0d8a06-133a-4d5d-a51c-5eec68151cad`::Granted request >Thread-2250::DEBUG::2012-08-30 13:20:50,761::task::817::TaskManager.Task::(resourceAcquired) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2250::DEBUG::2012-08-30 13:20:50,761::task::978::TaskManager.Task::(_decref) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::ref 1 aborting False >Thread-2250::INFO::2012-08-30 13:20:50,764::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2250::DEBUG::2012-08-30 13:20:50,764::task::1172::TaskManager.Task::(prepare) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2250::DEBUG::2012-08-30 13:20:50,764::task::588::TaskManager.Task::(_updateState) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::moving from state preparing -> state finished >Thread-2250::DEBUG::2012-08-30 13:20:50,764::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2250::DEBUG::2012-08-30 13:20:50,765::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2250::DEBUG::2012-08-30 13:20:50,765::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2250::DEBUG::2012-08-30 13:20:50,765::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2250::DEBUG::2012-08-30 13:20:50,765::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2250::DEBUG::2012-08-30 13:20:50,766::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2250::DEBUG::2012-08-30 13:20:50,766::task::978::TaskManager.Task::(_decref) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:20:50,937::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:20:53,842::task::588::TaskManager.Task::(_updateState) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:20:53,842::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:20:53,843::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`96a22286-cea5-4376-9d23-1e7440a98683`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:20:53,843::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:20:53,843::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:20:53,843::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`96a22286-cea5-4376-9d23-1e7440a98683`::Granted request >Thread-40::DEBUG::2012-08-30 13:20:53,844::task::817::TaskManager.Task::(resourceAcquired) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:20:53,844::task::978::TaskManager.Task::(_decref) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:20:53,846::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:20:53,848::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:20:53,849::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:20:53,849::task::1172::TaskManager.Task::(prepare) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:20:53,849::task::588::TaskManager.Task::(_updateState) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:20:53,850::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:20:53,850::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:20:53,850::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:20:53,850::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:20:53,851::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:20:53,851::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:20:53,851::task::978::TaskManager.Task::(_decref) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:20:53,851::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:53,852::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:53,852::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:20:53,853::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:20:55,938::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:20:56,600::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:20:58,854::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:00,939::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:00,940::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:21:03,608::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:21:03,855::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2254::DEBUG::2012-08-30 13:21:04,008::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2254::DEBUG::2012-08-30 13:21:04,009::task::588::TaskManager.Task::(_updateState) Task=`6e98d007-39da-4efa-83f2-ecbc46bf7070`::moving from state init -> state preparing >Thread-2254::INFO::2012-08-30 13:21:04,009::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2254::INFO::2012-08-30 13:21:04,009::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2254::DEBUG::2012-08-30 13:21:04,010::task::1172::TaskManager.Task::(prepare) Task=`6e98d007-39da-4efa-83f2-ecbc46bf7070`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2254::DEBUG::2012-08-30 13:21:04,010::task::588::TaskManager.Task::(_updateState) Task=`6e98d007-39da-4efa-83f2-ecbc46bf7070`::moving from state preparing -> state finished >Thread-2254::DEBUG::2012-08-30 13:21:04,010::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2254::DEBUG::2012-08-30 13:21:04,010::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2254::DEBUG::2012-08-30 13:21:04,010::task::978::TaskManager.Task::(_decref) Task=`6e98d007-39da-4efa-83f2-ecbc46bf7070`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:21:05,941::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2256::DEBUG::2012-08-30 13:21:07,341::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2256::DEBUG::2012-08-30 13:21:07,342::task::588::TaskManager.Task::(_updateState) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::moving from state init -> state preparing >Thread-2256::INFO::2012-08-30 13:21:07,342::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2256::DEBUG::2012-08-30 13:21:07,342::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c9ebba51-5be7-4010-ab19-f94d73d59c4c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2256::DEBUG::2012-08-30 13:21:07,343::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2256::DEBUG::2012-08-30 13:21:07,343::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2256::DEBUG::2012-08-30 13:21:07,343::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c9ebba51-5be7-4010-ab19-f94d73d59c4c`::Granted request >Thread-2256::DEBUG::2012-08-30 13:21:07,343::task::817::TaskManager.Task::(resourceAcquired) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2256::DEBUG::2012-08-30 13:21:07,344::task::978::TaskManager.Task::(_decref) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::ref 1 aborting False >Thread-2256::INFO::2012-08-30 13:21:07,347::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2256::DEBUG::2012-08-30 13:21:07,347::task::1172::TaskManager.Task::(prepare) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2256::DEBUG::2012-08-30 13:21:07,347::task::588::TaskManager.Task::(_updateState) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::moving from state preparing -> state finished >Thread-2256::DEBUG::2012-08-30 13:21:07,347::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2256::DEBUG::2012-08-30 13:21:07,349::task::978::TaskManager.Task::(_decref) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:21:08,856::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:21:08,857::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:10,942::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:21:13,858::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:15,943::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:15,943::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2258::DEBUG::2012-08-30 13:21:16,744::task::588::TaskManager.Task::(_updateState) Task=`bddd6ce6-845d-4571-9b90-1b20f174e9d5`::moving from state init -> state preparing >Thread-2258::INFO::2012-08-30 13:21:16,744::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2258::INFO::2012-08-30 13:21:16,745::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163388252258', 'lastCheck': 1346304071.583734, 'code': 0, 'valid': True}} >Thread-2258::DEBUG::2012-08-30 13:21:16,745::task::1172::TaskManager.Task::(prepare) Task=`bddd6ce6-845d-4571-9b90-1b20f174e9d5`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163388252258', 'lastCheck': 1346304071.583734, 'code': 0, 'valid': True}} >Thread-2258::DEBUG::2012-08-30 13:21:16,745::task::588::TaskManager.Task::(_updateState) Task=`bddd6ce6-845d-4571-9b90-1b20f174e9d5`::moving from state preparing -> state finished >Thread-2258::DEBUG::2012-08-30 13:21:16,745::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2258::DEBUG::2012-08-30 13:21:16,745::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2258::DEBUG::2012-08-30 13:21:16,746::task::978::TaskManager.Task::(_decref) Task=`bddd6ce6-845d-4571-9b90-1b20f174e9d5`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:21:18,859::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2259::DEBUG::2012-08-30 13:21:19,935::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2260::DEBUG::2012-08-30 13:21:19,936::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2259::DEBUG::2012-08-30 13:21:19,936::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-2260::DEBUG::2012-08-30 13:21:19,937::task::588::TaskManager.Task::(_updateState) Task=`1b37c8a5-78f6-4279-ad3d-b8171072e663`::moving from state init -> state preparing >Thread-2260::INFO::2012-08-30 13:21:19,938::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2260::INFO::2012-08-30 13:21:19,938::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2260::DEBUG::2012-08-30 13:21:19,939::task::1172::TaskManager.Task::(prepare) Task=`1b37c8a5-78f6-4279-ad3d-b8171072e663`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2260::DEBUG::2012-08-30 13:21:19,939::task::588::TaskManager.Task::(_updateState) Task=`1b37c8a5-78f6-4279-ad3d-b8171072e663`::moving from state preparing -> state finished >Thread-2260::DEBUG::2012-08-30 13:21:19,939::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2260::DEBUG::2012-08-30 13:21:19,939::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2260::DEBUG::2012-08-30 13:21:19,939::task::978::TaskManager.Task::(_decref) Task=`1b37c8a5-78f6-4279-ad3d-b8171072e663`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:21:20,944::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2261::DEBUG::2012-08-30 13:21:23,121::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2261::DEBUG::2012-08-30 13:21:23,121::task::588::TaskManager.Task::(_updateState) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::moving from state init -> state preparing >Thread-2261::INFO::2012-08-30 13:21:23,122::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2261::DEBUG::2012-08-30 13:21:23,122::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f06a4aad-502e-49ba-bc76-a592c4c9d9bc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2261::DEBUG::2012-08-30 13:21:23,122::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2261::DEBUG::2012-08-30 13:21:23,122::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2261::DEBUG::2012-08-30 13:21:23,123::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f06a4aad-502e-49ba-bc76-a592c4c9d9bc`::Granted request >Thread-2261::DEBUG::2012-08-30 13:21:23,123::task::817::TaskManager.Task::(resourceAcquired) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2261::DEBUG::2012-08-30 13:21:23,123::task::978::TaskManager.Task::(_decref) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::ref 1 aborting False >Thread-2261::INFO::2012-08-30 13:21:23,126::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2261::DEBUG::2012-08-30 13:21:23,126::task::1172::TaskManager.Task::(prepare) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2261::DEBUG::2012-08-30 13:21:23,127::task::588::TaskManager.Task::(_updateState) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::moving from state preparing -> state finished >Thread-2261::DEBUG::2012-08-30 13:21:23,127::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2261::DEBUG::2012-08-30 13:21:23,127::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2261::DEBUG::2012-08-30 13:21:23,127::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2261::DEBUG::2012-08-30 13:21:23,128::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2261::DEBUG::2012-08-30 13:21:23,128::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2261::DEBUG::2012-08-30 13:21:23,128::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2261::DEBUG::2012-08-30 13:21:23,128::task::978::TaskManager.Task::(_decref) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:21:23,861::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:21:23,861::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:25,945::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:21:26,627::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:21:28,862::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:21:30,946::task::588::TaskManager.Task::(_updateState) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:21:30,946::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:21:30,947::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a5826bcf-9087-49c5-b973-1e34eb3632a5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:21:30,947::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:21:30,947::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:21:30,947::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a5826bcf-9087-49c5-b973-1e34eb3632a5`::Granted request >Thread-104::DEBUG::2012-08-30 13:21:30,948::task::817::TaskManager.Task::(resourceAcquired) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:21:30,948::task::978::TaskManager.Task::(_decref) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:21:30,949::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:21:30,952::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:21:30,953::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:21:30,953::task::1172::TaskManager.Task::(prepare) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:21:30,953::task::588::TaskManager.Task::(_updateState) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:21:30,953::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:21:30,954::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:21:30,954::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:21:30,954::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:21:30,954::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:21:30,955::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:21:30,955::task::978::TaskManager.Task::(_decref) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:21:30,955::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:30,956::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:30,956::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:30,957::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:21:33,632::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:21:33,864::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:35,957::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2265::DEBUG::2012-08-30 13:21:36,328::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2265::DEBUG::2012-08-30 13:21:36,328::task::588::TaskManager.Task::(_updateState) Task=`3e775112-f3d1-4211-baf8-66bd1f92b12b`::moving from state init -> state preparing >Thread-2265::INFO::2012-08-30 13:21:36,329::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2265::INFO::2012-08-30 13:21:36,329::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2265::DEBUG::2012-08-30 13:21:36,329::task::1172::TaskManager.Task::(prepare) Task=`3e775112-f3d1-4211-baf8-66bd1f92b12b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2265::DEBUG::2012-08-30 13:21:36,329::task::588::TaskManager.Task::(_updateState) Task=`3e775112-f3d1-4211-baf8-66bd1f92b12b`::moving from state preparing -> state finished >Thread-2265::DEBUG::2012-08-30 13:21:36,330::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2265::DEBUG::2012-08-30 13:21:36,330::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2265::DEBUG::2012-08-30 13:21:36,330::task::978::TaskManager.Task::(_decref) Task=`3e775112-f3d1-4211-baf8-66bd1f92b12b`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:21:38,865::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:21:38,865::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2266::DEBUG::2012-08-30 13:21:39,197::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2266::DEBUG::2012-08-30 13:21:39,197::task::588::TaskManager.Task::(_updateState) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::moving from state init -> state preparing >Thread-2266::INFO::2012-08-30 13:21:39,197::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2266::DEBUG::2012-08-30 13:21:39,198::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6047fbad-ffe1-4f76-baae-3414da2366c6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2266::DEBUG::2012-08-30 13:21:39,198::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2266::DEBUG::2012-08-30 13:21:39,198::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2266::DEBUG::2012-08-30 13:21:39,198::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6047fbad-ffe1-4f76-baae-3414da2366c6`::Granted request >Thread-2266::DEBUG::2012-08-30 13:21:39,199::task::817::TaskManager.Task::(resourceAcquired) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2266::DEBUG::2012-08-30 13:21:39,199::task::978::TaskManager.Task::(_decref) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::ref 1 aborting False >Thread-2266::INFO::2012-08-30 13:21:39,202::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2266::DEBUG::2012-08-30 13:21:39,202::task::1172::TaskManager.Task::(prepare) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2266::DEBUG::2012-08-30 13:21:39,203::task::588::TaskManager.Task::(_updateState) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::moving from state preparing -> state finished >Thread-2266::DEBUG::2012-08-30 13:21:39,203::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2266::DEBUG::2012-08-30 13:21:39,203::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2266::DEBUG::2012-08-30 13:21:39,203::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2266::DEBUG::2012-08-30 13:21:39,203::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2266::DEBUG::2012-08-30 13:21:39,204::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2266::DEBUG::2012-08-30 13:21:39,204::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2266::DEBUG::2012-08-30 13:21:39,204::task::978::TaskManager.Task::(_decref) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:21:40,958::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:21:43,866::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:45,959::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:45,960::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2268::DEBUG::2012-08-30 13:21:46,182::task::588::TaskManager.Task::(_updateState) Task=`bfe4028f-757a-4a45-99f7-3846eb923b09`::moving from state init -> state preparing >Thread-2268::INFO::2012-08-30 13:21:46,182::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2268::INFO::2012-08-30 13:21:46,182::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162410736084', 'lastCheck': 1346304101.602519, 'code': 0, 'valid': True}} >Thread-2268::DEBUG::2012-08-30 13:21:46,182::task::1172::TaskManager.Task::(prepare) Task=`bfe4028f-757a-4a45-99f7-3846eb923b09`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162410736084', 'lastCheck': 1346304101.602519, 'code': 0, 'valid': True}} >Thread-2268::DEBUG::2012-08-30 13:21:46,183::task::588::TaskManager.Task::(_updateState) Task=`bfe4028f-757a-4a45-99f7-3846eb923b09`::moving from state preparing -> state finished >Thread-2268::DEBUG::2012-08-30 13:21:46,183::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2268::DEBUG::2012-08-30 13:21:46,183::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2268::DEBUG::2012-08-30 13:21:46,183::task::978::TaskManager.Task::(_decref) Task=`bfe4028f-757a-4a45-99f7-3846eb923b09`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:21:48,868::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2269::DEBUG::2012-08-30 13:21:49,660::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2269::DEBUG::2012-08-30 13:21:49,660::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:21:50,961::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2270::DEBUG::2012-08-30 13:21:53,640::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2270::DEBUG::2012-08-30 13:21:53,640::task::588::TaskManager.Task::(_updateState) Task=`0ec0d9ec-3650-4ec4-ad79-8abf10f4bf9a`::moving from state init -> state preparing >Thread-2270::INFO::2012-08-30 13:21:53,640::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2270::INFO::2012-08-30 13:21:53,641::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2270::DEBUG::2012-08-30 13:21:53,641::task::1172::TaskManager.Task::(prepare) Task=`0ec0d9ec-3650-4ec4-ad79-8abf10f4bf9a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2270::DEBUG::2012-08-30 13:21:53,641::task::588::TaskManager.Task::(_updateState) Task=`0ec0d9ec-3650-4ec4-ad79-8abf10f4bf9a`::moving from state preparing -> state finished >Thread-2270::DEBUG::2012-08-30 13:21:53,641::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2270::DEBUG::2012-08-30 13:21:53,642::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2270::DEBUG::2012-08-30 13:21:53,642::task::978::TaskManager.Task::(_decref) Task=`0ec0d9ec-3650-4ec4-ad79-8abf10f4bf9a`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 13:21:53,869::task::588::TaskManager.Task::(_updateState) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:21:53,869::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:21:53,870::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`37daa529-999e-4cea-a2e9-1b57673419dc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:21:53,870::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:21:53,870::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:21:53,870::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`37daa529-999e-4cea-a2e9-1b57673419dc`::Granted request >Thread-40::DEBUG::2012-08-30 13:21:53,871::task::817::TaskManager.Task::(resourceAcquired) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:21:53,871::task::978::TaskManager.Task::(_decref) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:21:53,873::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:21:53,875::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:21:53,876::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:21:53,876::task::1172::TaskManager.Task::(prepare) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:21:53,876::task::588::TaskManager.Task::(_updateState) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:21:53,878::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:21:53,878::task::978::TaskManager.Task::(_decref) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:21:53,878::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:21:53,879::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:21:53,879::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:21:53,880::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:21:55,962::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:21:56,654::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2272::DEBUG::2012-08-30 13:21:57,359::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2272::DEBUG::2012-08-30 13:21:57,360::task::588::TaskManager.Task::(_updateState) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::moving from state init -> state preparing >Thread-2272::INFO::2012-08-30 13:21:57,360::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2272::DEBUG::2012-08-30 13:21:57,360::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d5867a92-a3f1-4b04-aad2-45d46223ede0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2272::DEBUG::2012-08-30 13:21:57,360::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2272::DEBUG::2012-08-30 13:21:57,361::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2272::DEBUG::2012-08-30 13:21:57,361::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d5867a92-a3f1-4b04-aad2-45d46223ede0`::Granted request >Thread-2272::DEBUG::2012-08-30 13:21:57,361::task::817::TaskManager.Task::(resourceAcquired) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2272::DEBUG::2012-08-30 13:21:57,362::task::978::TaskManager.Task::(_decref) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::ref 1 aborting False >Thread-2272::INFO::2012-08-30 13:21:57,364::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2272::DEBUG::2012-08-30 13:21:57,365::task::1172::TaskManager.Task::(prepare) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2272::DEBUG::2012-08-30 13:21:57,365::task::588::TaskManager.Task::(_updateState) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::moving from state preparing -> state finished >Thread-2272::DEBUG::2012-08-30 13:21:57,365::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2272::DEBUG::2012-08-30 13:21:57,365::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2272::DEBUG::2012-08-30 13:21:57,366::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2272::DEBUG::2012-08-30 13:21:57,366::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2272::DEBUG::2012-08-30 13:21:57,366::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2272::DEBUG::2012-08-30 13:21:57,366::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2272::DEBUG::2012-08-30 13:21:57,367::task::978::TaskManager.Task::(_decref) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:21:58,881::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:00,963::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:00,963::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:22:03,660::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:22:03,882::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:05,964::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:08,883::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:08,884::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2275::DEBUG::2012-08-30 13:22:10,831::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2275::DEBUG::2012-08-30 13:22:10,832::task::588::TaskManager.Task::(_updateState) Task=`d18f7942-f50e-4910-bf77-1de0c7eb92a1`::moving from state init -> state preparing >Thread-2275::INFO::2012-08-30 13:22:10,832::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2275::INFO::2012-08-30 13:22:10,832::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2275::DEBUG::2012-08-30 13:22:10,833::task::1172::TaskManager.Task::(prepare) Task=`d18f7942-f50e-4910-bf77-1de0c7eb92a1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2275::DEBUG::2012-08-30 13:22:10,833::task::588::TaskManager.Task::(_updateState) Task=`d18f7942-f50e-4910-bf77-1de0c7eb92a1`::moving from state preparing -> state finished >Thread-2275::DEBUG::2012-08-30 13:22:10,833::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2275::DEBUG::2012-08-30 13:22:10,833::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2275::DEBUG::2012-08-30 13:22:10,834::task::978::TaskManager.Task::(_decref) Task=`d18f7942-f50e-4910-bf77-1de0c7eb92a1`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:22:10,965::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:13,885::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2277::DEBUG::2012-08-30 13:22:14,213::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2277::DEBUG::2012-08-30 13:22:14,214::task::588::TaskManager.Task::(_updateState) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::moving from state init -> state preparing >Thread-2277::INFO::2012-08-30 13:22:14,214::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2277::DEBUG::2012-08-30 13:22:14,214::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9ac115c3-5e59-4535-a6bb-aabc1151e020`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2277::DEBUG::2012-08-30 13:22:14,214::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2277::DEBUG::2012-08-30 13:22:14,215::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2277::DEBUG::2012-08-30 13:22:14,215::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9ac115c3-5e59-4535-a6bb-aabc1151e020`::Granted request >Thread-2277::DEBUG::2012-08-30 13:22:14,215::task::817::TaskManager.Task::(resourceAcquired) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2277::DEBUG::2012-08-30 13:22:14,216::task::978::TaskManager.Task::(_decref) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::ref 1 aborting False >Thread-2277::INFO::2012-08-30 13:22:14,218::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2277::DEBUG::2012-08-30 13:22:14,219::task::1172::TaskManager.Task::(prepare) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2277::DEBUG::2012-08-30 13:22:14,219::task::588::TaskManager.Task::(_updateState) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::moving from state preparing -> state finished >Thread-2277::DEBUG::2012-08-30 13:22:14,219::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2277::DEBUG::2012-08-30 13:22:14,219::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2277::DEBUG::2012-08-30 13:22:14,220::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2277::DEBUG::2012-08-30 13:22:14,220::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2277::DEBUG::2012-08-30 13:22:14,220::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2277::DEBUG::2012-08-30 13:22:14,220::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2277::DEBUG::2012-08-30 13:22:14,221::task::978::TaskManager.Task::(_decref) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::ref 0 aborting False >Thread-2278::DEBUG::2012-08-30 13:22:15,843::task::588::TaskManager.Task::(_updateState) Task=`0df545c5-2d73-48b1-b044-a17d0739e2cf`::moving from state init -> state preparing >Thread-2278::INFO::2012-08-30 13:22:15,843::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2278::INFO::2012-08-30 13:22:15,843::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162982940674', 'lastCheck': 1346304131.6231959, 'code': 0, 'valid': True}} >Thread-2278::DEBUG::2012-08-30 13:22:15,843::task::1172::TaskManager.Task::(prepare) Task=`0df545c5-2d73-48b1-b044-a17d0739e2cf`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162982940674', 'lastCheck': 1346304131.6231959, 'code': 0, 'valid': True}} >Thread-2278::DEBUG::2012-08-30 13:22:15,844::task::588::TaskManager.Task::(_updateState) Task=`0df545c5-2d73-48b1-b044-a17d0739e2cf`::moving from state preparing -> state finished >Thread-2278::DEBUG::2012-08-30 13:22:15,844::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2278::DEBUG::2012-08-30 13:22:15,844::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2278::DEBUG::2012-08-30 13:22:15,844::task::978::TaskManager.Task::(_decref) Task=`0df545c5-2d73-48b1-b044-a17d0739e2cf`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:22:15,966::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:15,966::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:18,886::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2279::DEBUG::2012-08-30 13:22:19,052::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2279::DEBUG::2012-08-30 13:22:19,053::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:22:20,967::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:23,887::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:23,888::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:25,968::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:22:26,682::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-2281::DEBUG::2012-08-30 13:22:27,479::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2281::DEBUG::2012-08-30 13:22:27,479::task::588::TaskManager.Task::(_updateState) Task=`a6c24215-778a-4f56-971d-5fc9c00b3831`::moving from state init -> state preparing >Thread-2281::INFO::2012-08-30 13:22:27,480::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2281::INFO::2012-08-30 13:22:27,480::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2281::DEBUG::2012-08-30 13:22:27,480::task::1172::TaskManager.Task::(prepare) Task=`a6c24215-778a-4f56-971d-5fc9c00b3831`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2281::DEBUG::2012-08-30 13:22:27,480::task::588::TaskManager.Task::(_updateState) Task=`a6c24215-778a-4f56-971d-5fc9c00b3831`::moving from state preparing -> state finished >Thread-2281::DEBUG::2012-08-30 13:22:27,481::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2281::DEBUG::2012-08-30 13:22:27,481::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2281::DEBUG::2012-08-30 13:22:27,481::task::978::TaskManager.Task::(_decref) Task=`a6c24215-778a-4f56-971d-5fc9c00b3831`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:22:28,889::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2283::DEBUG::2012-08-30 13:22:30,618::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2283::DEBUG::2012-08-30 13:22:30,619::task::588::TaskManager.Task::(_updateState) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::moving from state init -> state preparing >Thread-2283::INFO::2012-08-30 13:22:30,619::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2283::DEBUG::2012-08-30 13:22:30,620::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`26a150fd-2e26-4e67-8ec3-7b277017f930`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2283::DEBUG::2012-08-30 13:22:30,620::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2283::DEBUG::2012-08-30 13:22:30,620::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2283::DEBUG::2012-08-30 13:22:30,620::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`26a150fd-2e26-4e67-8ec3-7b277017f930`::Granted request >Thread-2283::DEBUG::2012-08-30 13:22:30,621::task::817::TaskManager.Task::(resourceAcquired) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2283::DEBUG::2012-08-30 13:22:30,621::task::978::TaskManager.Task::(_decref) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::ref 1 aborting False >Thread-2283::INFO::2012-08-30 13:22:30,624::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2283::DEBUG::2012-08-30 13:22:30,624::task::1172::TaskManager.Task::(prepare) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2283::DEBUG::2012-08-30 13:22:30,624::task::588::TaskManager.Task::(_updateState) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::moving from state preparing -> state finished >Thread-2283::DEBUG::2012-08-30 13:22:30,625::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2283::DEBUG::2012-08-30 13:22:30,625::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2283::DEBUG::2012-08-30 13:22:30,625::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2283::DEBUG::2012-08-30 13:22:30,625::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2283::DEBUG::2012-08-30 13:22:30,626::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2283::DEBUG::2012-08-30 13:22:30,626::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2283::DEBUG::2012-08-30 13:22:30,626::task::978::TaskManager.Task::(_decref) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::ref 0 aborting False >Thread-104::DEBUG::2012-08-30 13:22:30,969::task::588::TaskManager.Task::(_updateState) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:22:30,969::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:22:30,970::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`09cc1736-a696-4ef4-b285-6b50d185a4be`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:22:30,970::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:22:30,970::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:22:30,970::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`09cc1736-a696-4ef4-b285-6b50d185a4be`::Granted request >Thread-104::DEBUG::2012-08-30 13:22:30,971::task::817::TaskManager.Task::(resourceAcquired) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:22:30,971::task::978::TaskManager.Task::(_decref) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:22:30,972::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:22:30,975::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:22:30,976::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:22:30,976::task::1172::TaskManager.Task::(prepare) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:22:30,976::task::588::TaskManager.Task::(_updateState) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:22:30,976::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:22:30,978::task::978::TaskManager.Task::(_decref) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:22:30,978::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:30,979::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:30,979::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:30,979::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:22:33,688::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:22:33,890::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:35,980::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:38,892::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:38,892::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:40,981::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2286::DEBUG::2012-08-30 13:22:43,307::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2286::DEBUG::2012-08-30 13:22:43,308::task::588::TaskManager.Task::(_updateState) Task=`0da26833-98dc-422c-8a87-13cca0acaa5e`::moving from state init -> state preparing >Thread-2286::INFO::2012-08-30 13:22:43,308::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2286::INFO::2012-08-30 13:22:43,308::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2286::DEBUG::2012-08-30 13:22:43,308::task::1172::TaskManager.Task::(prepare) Task=`0da26833-98dc-422c-8a87-13cca0acaa5e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2286::DEBUG::2012-08-30 13:22:43,309::task::588::TaskManager.Task::(_updateState) Task=`0da26833-98dc-422c-8a87-13cca0acaa5e`::moving from state preparing -> state finished >Thread-2286::DEBUG::2012-08-30 13:22:43,309::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2286::DEBUG::2012-08-30 13:22:43,309::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2286::DEBUG::2012-08-30 13:22:43,309::task::978::TaskManager.Task::(_decref) Task=`0da26833-98dc-422c-8a87-13cca0acaa5e`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:22:43,893::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2287::DEBUG::2012-08-30 13:22:45,148::task::588::TaskManager.Task::(_updateState) Task=`a4755cba-2528-42f2-a828-4eb672dbcb01`::moving from state init -> state preparing >Thread-2287::INFO::2012-08-30 13:22:45,148::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2287::INFO::2012-08-30 13:22:45,149::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346304161.6421461, 'code': 0, 'valid': True}} >Thread-2287::DEBUG::2012-08-30 13:22:45,149::task::1172::TaskManager.Task::(prepare) Task=`a4755cba-2528-42f2-a828-4eb672dbcb01`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346304161.6421461, 'code': 0, 'valid': True}} >Thread-2287::DEBUG::2012-08-30 13:22:45,149::task::588::TaskManager.Task::(_updateState) Task=`a4755cba-2528-42f2-a828-4eb672dbcb01`::moving from state preparing -> state finished >Thread-2287::DEBUG::2012-08-30 13:22:45,149::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2287::DEBUG::2012-08-30 13:22:45,150::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2287::DEBUG::2012-08-30 13:22:45,150::task::978::TaskManager.Task::(_decref) Task=`a4755cba-2528-42f2-a828-4eb672dbcb01`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:22:45,982::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:45,983::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2288::DEBUG::2012-08-30 13:22:46,722::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2288::DEBUG::2012-08-30 13:22:46,723::task::588::TaskManager.Task::(_updateState) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::moving from state init -> state preparing >Thread-2288::INFO::2012-08-30 13:22:46,723::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2288::DEBUG::2012-08-30 13:22:46,724::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`92da28b6-5e2d-4c0f-9138-e4a007d8ea8f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2288::DEBUG::2012-08-30 13:22:46,724::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2288::DEBUG::2012-08-30 13:22:46,724::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2288::DEBUG::2012-08-30 13:22:46,724::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`92da28b6-5e2d-4c0f-9138-e4a007d8ea8f`::Granted request >Thread-2288::DEBUG::2012-08-30 13:22:46,725::task::817::TaskManager.Task::(resourceAcquired) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2288::DEBUG::2012-08-30 13:22:46,725::task::978::TaskManager.Task::(_decref) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::ref 1 aborting False >Thread-2288::INFO::2012-08-30 13:22:46,728::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2288::DEBUG::2012-08-30 13:22:46,728::task::1172::TaskManager.Task::(prepare) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2288::DEBUG::2012-08-30 13:22:46,728::task::588::TaskManager.Task::(_updateState) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::moving from state preparing -> state finished >Thread-2288::DEBUG::2012-08-30 13:22:46,728::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2288::DEBUG::2012-08-30 13:22:46,729::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2288::DEBUG::2012-08-30 13:22:46,729::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2288::DEBUG::2012-08-30 13:22:46,729::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2288::DEBUG::2012-08-30 13:22:46,729::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2288::DEBUG::2012-08-30 13:22:46,730::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2288::DEBUG::2012-08-30 13:22:46,730::task::978::TaskManager.Task::(_decref) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::ref 0 aborting False >Thread-2289::DEBUG::2012-08-30 13:22:48,301::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2289::DEBUG::2012-08-30 13:22:48,302::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:22:48,895::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:50,984::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::DEBUG::2012-08-30 13:22:53,896::task::588::TaskManager.Task::(_updateState) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:22:53,896::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:22:53,897::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4fc1ced7-b915-492d-bca4-25d7c9367a46`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:22:53,897::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:22:53,897::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:22:53,897::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4fc1ced7-b915-492d-bca4-25d7c9367a46`::Granted request >Thread-40::DEBUG::2012-08-30 13:22:53,898::task::817::TaskManager.Task::(resourceAcquired) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:22:53,898::task::978::TaskManager.Task::(_decref) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:22:53,900::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:22:53,902::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:22:53,903::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:22:53,903::task::1172::TaskManager.Task::(prepare) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:22:53,903::task::588::TaskManager.Task::(_updateState) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:22:53,903::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:22:53,904::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:22:53,904::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:22:53,904::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:22:53,904::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:22:53,905::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:22:53,905::task::978::TaskManager.Task::(_decref) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:22:53,905::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:53,906::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:53,906::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:22:53,907::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:22:55,985::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:22:56,710::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:22:58,908::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2292::DEBUG::2012-08-30 13:22:59,949::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2292::DEBUG::2012-08-30 13:22:59,950::task::588::TaskManager.Task::(_updateState) Task=`11e1eb14-da86-4518-8410-66b0d21273c7`::moving from state init -> state preparing >Thread-2292::INFO::2012-08-30 13:22:59,950::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2292::INFO::2012-08-30 13:22:59,950::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2292::DEBUG::2012-08-30 13:22:59,950::task::1172::TaskManager.Task::(prepare) Task=`11e1eb14-da86-4518-8410-66b0d21273c7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2292::DEBUG::2012-08-30 13:22:59,951::task::588::TaskManager.Task::(_updateState) Task=`11e1eb14-da86-4518-8410-66b0d21273c7`::moving from state preparing -> state finished >Thread-2292::DEBUG::2012-08-30 13:22:59,951::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2292::DEBUG::2012-08-30 13:22:59,951::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2292::DEBUG::2012-08-30 13:22:59,951::task::978::TaskManager.Task::(_decref) Task=`11e1eb14-da86-4518-8410-66b0d21273c7`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:23:00,986::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:00,986::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2293::DEBUG::2012-08-30 13:23:03,146::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2293::DEBUG::2012-08-30 13:23:03,147::task::588::TaskManager.Task::(_updateState) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::moving from state init -> state preparing >Thread-2293::INFO::2012-08-30 13:23:03,147::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2293::DEBUG::2012-08-30 13:23:03,147::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5950726f-36e5-4a47-8c47-5a3f20a0f673`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2293::DEBUG::2012-08-30 13:23:03,148::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2293::DEBUG::2012-08-30 13:23:03,148::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2293::DEBUG::2012-08-30 13:23:03,148::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5950726f-36e5-4a47-8c47-5a3f20a0f673`::Granted request >Thread-2293::DEBUG::2012-08-30 13:23:03,148::task::817::TaskManager.Task::(resourceAcquired) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2293::DEBUG::2012-08-30 13:23:03,149::task::978::TaskManager.Task::(_decref) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::ref 1 aborting False >Thread-2293::INFO::2012-08-30 13:23:03,152::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2293::DEBUG::2012-08-30 13:23:03,152::task::1172::TaskManager.Task::(prepare) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2293::DEBUG::2012-08-30 13:23:03,152::task::588::TaskManager.Task::(_updateState) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::moving from state preparing -> state finished >Thread-2293::DEBUG::2012-08-30 13:23:03,152::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2293::DEBUG::2012-08-30 13:23:03,153::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2293::DEBUG::2012-08-30 13:23:03,153::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2293::DEBUG::2012-08-30 13:23:03,153::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2293::DEBUG::2012-08-30 13:23:03,153::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2293::DEBUG::2012-08-30 13:23:03,154::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2293::DEBUG::2012-08-30 13:23:03,154::task::978::TaskManager.Task::(_decref) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:23:03,717::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:23:03,909::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:05,987::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:08,910::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:08,911::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:10,988::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2296::DEBUG::2012-08-30 13:23:13,456::task::588::TaskManager.Task::(_updateState) Task=`c3d01d39-8484-480f-b778-0f8b289c0e76`::moving from state init -> state preparing >Thread-2296::INFO::2012-08-30 13:23:13,457::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2296::INFO::2012-08-30 13:23:13,457::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346304191.6629989, 'code': 0, 'valid': True}} >Thread-2296::DEBUG::2012-08-30 13:23:13,457::task::1172::TaskManager.Task::(prepare) Task=`c3d01d39-8484-480f-b778-0f8b289c0e76`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346304191.6629989, 'code': 0, 'valid': True}} >Thread-2296::DEBUG::2012-08-30 13:23:13,457::task::588::TaskManager.Task::(_updateState) Task=`c3d01d39-8484-480f-b778-0f8b289c0e76`::moving from state preparing -> state finished >Thread-2296::DEBUG::2012-08-30 13:23:13,458::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2296::DEBUG::2012-08-30 13:23:13,458::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2296::DEBUG::2012-08-30 13:23:13,458::task::978::TaskManager.Task::(_decref) Task=`c3d01d39-8484-480f-b778-0f8b289c0e76`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:23:13,912::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:15,989::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:15,989::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2298::DEBUG::2012-08-30 13:23:16,388::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2297::DEBUG::2012-08-30 13:23:16,388::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2298::DEBUG::2012-08-30 13:23:16,389::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-2297::DEBUG::2012-08-30 13:23:16,389::task::588::TaskManager.Task::(_updateState) Task=`e3c943a9-f675-46c7-8811-db294e5cab73`::moving from state init -> state preparing >Thread-2297::INFO::2012-08-30 13:23:16,390::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2297::INFO::2012-08-30 13:23:16,391::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2297::DEBUG::2012-08-30 13:23:16,391::task::1172::TaskManager.Task::(prepare) Task=`e3c943a9-f675-46c7-8811-db294e5cab73`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2297::DEBUG::2012-08-30 13:23:16,391::task::588::TaskManager.Task::(_updateState) Task=`e3c943a9-f675-46c7-8811-db294e5cab73`::moving from state preparing -> state finished >Thread-2297::DEBUG::2012-08-30 13:23:16,392::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2297::DEBUG::2012-08-30 13:23:16,392::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2297::DEBUG::2012-08-30 13:23:16,392::task::978::TaskManager.Task::(_decref) Task=`e3c943a9-f675-46c7-8811-db294e5cab73`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:23:18,913::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2299::DEBUG::2012-08-30 13:23:19,553::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2299::DEBUG::2012-08-30 13:23:19,553::task::588::TaskManager.Task::(_updateState) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::moving from state init -> state preparing >Thread-2299::INFO::2012-08-30 13:23:19,553::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2299::DEBUG::2012-08-30 13:23:19,554::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ab67c026-2633-4ded-8711-861c4eeece2e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2299::DEBUG::2012-08-30 13:23:19,555::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2299::DEBUG::2012-08-30 13:23:19,555::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2299::DEBUG::2012-08-30 13:23:19,555::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ab67c026-2633-4ded-8711-861c4eeece2e`::Granted request >Thread-2299::DEBUG::2012-08-30 13:23:19,555::task::817::TaskManager.Task::(resourceAcquired) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2299::DEBUG::2012-08-30 13:23:19,556::task::978::TaskManager.Task::(_decref) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::ref 1 aborting False >Thread-2299::INFO::2012-08-30 13:23:19,559::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2299::DEBUG::2012-08-30 13:23:19,559::task::1172::TaskManager.Task::(prepare) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2299::DEBUG::2012-08-30 13:23:19,559::task::588::TaskManager.Task::(_updateState) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::moving from state preparing -> state finished >Thread-2299::DEBUG::2012-08-30 13:23:19,559::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2299::DEBUG::2012-08-30 13:23:19,561::task::978::TaskManager.Task::(_decref) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:23:20,990::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:23,915::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:23,915::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:25,991::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:23:26,740::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:23:28,916::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::DEBUG::2012-08-30 13:23:30,992::task::588::TaskManager.Task::(_updateState) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::moving from state init -> state preparing >Thread-104::INFO::2012-08-30 13:23:30,993::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None) >Thread-104::DEBUG::2012-08-30 13:23:30,993::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ecbb9aef-8e55-4b0e-be31-90f83d2674a4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-104::DEBUG::2012-08-30 13:23:30,993::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-104::DEBUG::2012-08-30 13:23:30,993::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-104::DEBUG::2012-08-30 13:23:30,994::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ecbb9aef-8e55-4b0e-be31-90f83d2674a4`::Granted request >Thread-104::DEBUG::2012-08-30 13:23:30,994::task::817::TaskManager.Task::(resourceAcquired) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-104::DEBUG::2012-08-30 13:23:30,994::task::978::TaskManager.Task::(_decref) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::ref 1 aborting False >Thread-104::DEBUG::2012-08-30 13:23:30,996::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::DEBUG::2012-08-30 13:23:30,998::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361 >Thread-104::INFO::2012-08-30 13:23:30,999::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:23:30,999::task::1172::TaskManager.Task::(prepare) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::finished: {'truesize': '4096', 'apparentsize': '8589934592'} >Thread-104::DEBUG::2012-08-30 13:23:30,999::task::588::TaskManager.Task::(_updateState) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::moving from state preparing -> state finished >Thread-104::DEBUG::2012-08-30 13:23:31,000::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-104::DEBUG::2012-08-30 13:23:31,000::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-104::DEBUG::2012-08-30 13:23:31,000::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-104::DEBUG::2012-08-30 13:23:31,000::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-104::DEBUG::2012-08-30 13:23:31,001::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-104::DEBUG::2012-08-30 13:23:31,001::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-104::DEBUG::2012-08-30 13:23:31,001::task::978::TaskManager.Task::(_decref) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:23:31,002::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:31,002::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:31,002::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:31,003::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2303::DEBUG::2012-08-30 13:23:32,724::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2303::DEBUG::2012-08-30 13:23:32,725::task::588::TaskManager.Task::(_updateState) Task=`6f798e71-d106-46db-a85e-60af2ab50a43`::moving from state init -> state preparing >Thread-2303::INFO::2012-08-30 13:23:32,725::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2303::INFO::2012-08-30 13:23:32,725::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2303::DEBUG::2012-08-30 13:23:32,725::task::1172::TaskManager.Task::(prepare) Task=`6f798e71-d106-46db-a85e-60af2ab50a43`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2303::DEBUG::2012-08-30 13:23:32,726::task::588::TaskManager.Task::(_updateState) Task=`6f798e71-d106-46db-a85e-60af2ab50a43`::moving from state preparing -> state finished >Thread-2303::DEBUG::2012-08-30 13:23:32,726::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2303::DEBUG::2012-08-30 13:23:32,726::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2303::DEBUG::2012-08-30 13:23:32,726::task::978::TaskManager.Task::(_decref) Task=`6f798e71-d106-46db-a85e-60af2ab50a43`::ref 0 aborting False >VM Channels Listener::DEBUG::2012-08-30 13:23:33,745::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:23:33,918::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2304::DEBUG::2012-08-30 13:23:35,870::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2304::DEBUG::2012-08-30 13:23:35,871::task::588::TaskManager.Task::(_updateState) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::moving from state init -> state preparing >Thread-2304::INFO::2012-08-30 13:23:35,871::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2304::DEBUG::2012-08-30 13:23:35,872::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1c36469c-e257-421b-b13a-8a29bf5d6509`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2304::DEBUG::2012-08-30 13:23:35,872::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2304::DEBUG::2012-08-30 13:23:35,872::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2304::DEBUG::2012-08-30 13:23:35,873::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1c36469c-e257-421b-b13a-8a29bf5d6509`::Granted request >Thread-2304::DEBUG::2012-08-30 13:23:35,873::task::817::TaskManager.Task::(resourceAcquired) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2304::DEBUG::2012-08-30 13:23:35,873::task::978::TaskManager.Task::(_decref) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::ref 1 aborting False >Thread-2304::INFO::2012-08-30 13:23:35,876::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2304::DEBUG::2012-08-30 13:23:35,876::task::1172::TaskManager.Task::(prepare) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2304::DEBUG::2012-08-30 13:23:35,877::task::588::TaskManager.Task::(_updateState) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::moving from state preparing -> state finished >Thread-2304::DEBUG::2012-08-30 13:23:35,877::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2304::DEBUG::2012-08-30 13:23:35,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2304::DEBUG::2012-08-30 13:23:35,877::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2304::DEBUG::2012-08-30 13:23:35,878::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2304::DEBUG::2012-08-30 13:23:35,878::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2304::DEBUG::2012-08-30 13:23:35,878::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2304::DEBUG::2012-08-30 13:23:35,878::task::978::TaskManager.Task::(_decref) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:23:36,004::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:38,919::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:38,919::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:41,005::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2306::DEBUG::2012-08-30 13:23:41,027::task::588::TaskManager.Task::(_updateState) Task=`15b626d5-3fd0-462e-9e74-8d411fef50d2`::moving from state init -> state preparing >Thread-2306::INFO::2012-08-30 13:23:41,028::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2306::INFO::2012-08-30 13:23:41,028::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164413452148', 'lastCheck': 1346304211.6756511, 'code': 0, 'valid': True}} >Thread-2306::DEBUG::2012-08-30 13:23:41,028::task::1172::TaskManager.Task::(prepare) Task=`15b626d5-3fd0-462e-9e74-8d411fef50d2`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164413452148', 'lastCheck': 1346304211.6756511, 'code': 0, 'valid': True}} >Thread-2306::DEBUG::2012-08-30 13:23:41,028::task::588::TaskManager.Task::(_updateState) Task=`15b626d5-3fd0-462e-9e74-8d411fef50d2`::moving from state preparing -> state finished >Thread-2306::DEBUG::2012-08-30 13:23:41,028::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2306::DEBUG::2012-08-30 13:23:41,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2306::DEBUG::2012-08-30 13:23:41,029::task::978::TaskManager.Task::(_decref) Task=`15b626d5-3fd0-462e-9e74-8d411fef50d2`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:23:43,921::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2307::DEBUG::2012-08-30 13:23:44,176::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2307::DEBUG::2012-08-30 13:23:44,176::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-104::ERROR::2012-08-30 13:23:46,006::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:46,006::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:48,922::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2308::DEBUG::2012-08-30 13:23:49,056::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2308::DEBUG::2012-08-30 13:23:49,057::task::588::TaskManager.Task::(_updateState) Task=`aa4847c9-629a-4a77-b4a5-a91b8ad661aa`::moving from state init -> state preparing >Thread-2308::INFO::2012-08-30 13:23:49,057::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2308::INFO::2012-08-30 13:23:49,057::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2308::DEBUG::2012-08-30 13:23:49,058::task::1172::TaskManager.Task::(prepare) Task=`aa4847c9-629a-4a77-b4a5-a91b8ad661aa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2308::DEBUG::2012-08-30 13:23:49,058::task::588::TaskManager.Task::(_updateState) Task=`aa4847c9-629a-4a77-b4a5-a91b8ad661aa`::moving from state preparing -> state finished >Thread-2308::DEBUG::2012-08-30 13:23:49,058::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2308::DEBUG::2012-08-30 13:23:49,059::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2308::DEBUG::2012-08-30 13:23:49,059::task::978::TaskManager.Task::(_decref) Task=`aa4847c9-629a-4a77-b4a5-a91b8ad661aa`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:23:51,007::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2310::DEBUG::2012-08-30 13:23:52,204::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2310::DEBUG::2012-08-30 13:23:52,205::task::588::TaskManager.Task::(_updateState) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::moving from state init -> state preparing >Thread-2310::INFO::2012-08-30 13:23:52,205::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2310::DEBUG::2012-08-30 13:23:52,205::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`50903081-43f4-4a53-9177-6c080584534c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2310::DEBUG::2012-08-30 13:23:52,205::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2310::DEBUG::2012-08-30 13:23:52,206::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2310::DEBUG::2012-08-30 13:23:52,206::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`50903081-43f4-4a53-9177-6c080584534c`::Granted request >Thread-2310::DEBUG::2012-08-30 13:23:52,206::task::817::TaskManager.Task::(resourceAcquired) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2310::DEBUG::2012-08-30 13:23:52,207::task::978::TaskManager.Task::(_decref) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::ref 1 aborting False >Thread-2310::INFO::2012-08-30 13:23:52,209::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2310::DEBUG::2012-08-30 13:23:52,210::task::1172::TaskManager.Task::(prepare) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2310::DEBUG::2012-08-30 13:23:52,210::task::588::TaskManager.Task::(_updateState) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::moving from state preparing -> state finished >Thread-2310::DEBUG::2012-08-30 13:23:52,210::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2310::DEBUG::2012-08-30 13:23:52,210::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2310::DEBUG::2012-08-30 13:23:52,211::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2310::DEBUG::2012-08-30 13:23:52,211::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2310::DEBUG::2012-08-30 13:23:52,211::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2310::DEBUG::2012-08-30 13:23:52,211::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2310::DEBUG::2012-08-30 13:23:52,212::task::978::TaskManager.Task::(_decref) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::ref 0 aborting False >Thread-40::DEBUG::2012-08-30 13:23:53,923::task::588::TaskManager.Task::(_updateState) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::moving from state init -> state preparing >Thread-40::INFO::2012-08-30 13:23:53,923::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None) >Thread-40::DEBUG::2012-08-30 13:23:53,924::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9d7aabef-71e6-47bb-aea1-402faec64d10`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-40::DEBUG::2012-08-30 13:23:53,924::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared' >Thread-40::DEBUG::2012-08-30 13:23:53,924::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user) >Thread-40::DEBUG::2012-08-30 13:23:53,924::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9d7aabef-71e6-47bb-aea1-402faec64d10`::Granted request >Thread-40::DEBUG::2012-08-30 13:23:53,925::task::817::TaskManager.Task::(resourceAcquired) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared) >Thread-40::DEBUG::2012-08-30 13:23:53,925::task::978::TaskManager.Task::(_decref) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::ref 1 aborting False >Thread-40::DEBUG::2012-08-30 13:23:53,927::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::DEBUG::2012-08-30 13:23:53,929::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f >Thread-40::INFO::2012-08-30 13:23:53,930::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:23:53,930::task::1172::TaskManager.Task::(prepare) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'} >Thread-40::DEBUG::2012-08-30 13:23:53,930::task::588::TaskManager.Task::(_updateState) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::moving from state preparing -> state finished >Thread-40::DEBUG::2012-08-30 13:23:53,931::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>} >Thread-40::DEBUG::2012-08-30 13:23:53,931::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2012-08-30 13:23:53,931::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' >Thread-40::DEBUG::2012-08-30 13:23:53,931::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users) >Thread-40::DEBUG::2012-08-30 13:23:53,932::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it. >Thread-40::DEBUG::2012-08-30 13:23:53,932::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records. >Thread-40::DEBUG::2012-08-30 13:23:53,932::task::978::TaskManager.Task::(_decref) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:23:53,933::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:53,933::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk > diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats > if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:53,933::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency > libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT) > File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand > if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed') >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:23:53,934::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:23:56,008::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:23:56,764::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60. >Thread-40::ERROR::2012-08-30 13:23:58,935::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:24:01,009::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:24:01,010::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >VM Channels Listener::DEBUG::2012-08-30 13:24:03,770::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63. >Thread-40::ERROR::2012-08-30 13:24:03,936::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2314::DEBUG::2012-08-30 13:24:05,390::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2314::DEBUG::2012-08-30 13:24:05,391::task::588::TaskManager.Task::(_updateState) Task=`794f26d8-b8e7-49ac-a808-89b26836eb9f`::moving from state init -> state preparing >Thread-2314::INFO::2012-08-30 13:24:05,391::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2314::INFO::2012-08-30 13:24:05,391::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2314::DEBUG::2012-08-30 13:24:05,392::task::1172::TaskManager.Task::(prepare) Task=`794f26d8-b8e7-49ac-a808-89b26836eb9f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}} >Thread-2314::DEBUG::2012-08-30 13:24:05,392::task::588::TaskManager.Task::(_updateState) Task=`794f26d8-b8e7-49ac-a808-89b26836eb9f`::moving from state preparing -> state finished >Thread-2314::DEBUG::2012-08-30 13:24:05,392::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2314::DEBUG::2012-08-30 13:24:05,392::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2314::DEBUG::2012-08-30 13:24:05,393::task::978::TaskManager.Task::(_decref) Task=`794f26d8-b8e7-49ac-a808-89b26836eb9f`::ref 0 aborting False >Thread-104::ERROR::2012-08-30 13:24:06,010::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2315::DEBUG::2012-08-30 13:24:08,540::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12] >Thread-2316::DEBUG::2012-08-30 13:24:08,540::task::588::TaskManager.Task::(_updateState) Task=`b3a65101-83a8-47dc-ad17-6d460014534d`::moving from state init -> state preparing >Thread-2315::DEBUG::2012-08-30 13:24:08,541::task::588::TaskManager.Task::(_updateState) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::moving from state init -> state preparing >Thread-2316::INFO::2012-08-30 13:24:08,541::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2315::INFO::2012-08-30 13:24:08,541::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None) >Thread-2316::INFO::2012-08-30 13:24:08,541::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178599357605', 'lastCheck': 1346304241.6961839, 'code': 0, 'valid': True}} >Thread-2315::DEBUG::2012-08-30 13:24:08,542::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2cff57be-a476-488b-8120-2acf937cb3e1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2316::DEBUG::2012-08-30 13:24:08,542::task::1172::TaskManager.Task::(prepare) Task=`b3a65101-83a8-47dc-ad17-6d460014534d`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178599357605', 'lastCheck': 1346304241.6961839, 'code': 0, 'valid': True}} >Thread-2315::DEBUG::2012-08-30 13:24:08,542::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared' >Thread-2316::DEBUG::2012-08-30 13:24:08,542::task::588::TaskManager.Task::(_updateState) Task=`b3a65101-83a8-47dc-ad17-6d460014534d`::moving from state preparing -> state finished >Thread-2315::DEBUG::2012-08-30 13:24:08,543::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user) >Thread-2316::DEBUG::2012-08-30 13:24:08,543::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2315::DEBUG::2012-08-30 13:24:08,543::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2cff57be-a476-488b-8120-2acf937cb3e1`::Granted request >Thread-2316::DEBUG::2012-08-30 13:24:08,543::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2315::DEBUG::2012-08-30 13:24:08,543::task::817::TaskManager.Task::(resourceAcquired) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared) >Thread-2316::DEBUG::2012-08-30 13:24:08,544::task::978::TaskManager.Task::(_decref) Task=`b3a65101-83a8-47dc-ad17-6d460014534d`::ref 0 aborting False >Thread-2315::DEBUG::2012-08-30 13:24:08,544::task::978::TaskManager.Task::(_decref) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::ref 1 aborting False >Thread-2315::INFO::2012-08-30 13:24:08,549::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2315::DEBUG::2012-08-30 13:24:08,549::task::1172::TaskManager.Task::(prepare) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}} >Thread-2315::DEBUG::2012-08-30 13:24:08,549::task::588::TaskManager.Task::(_updateState) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::moving from state preparing -> state finished >Thread-2315::DEBUG::2012-08-30 13:24:08,549::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>} >Thread-2315::DEBUG::2012-08-30 13:24:08,550::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2315::DEBUG::2012-08-30 13:24:08,550::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' >Thread-2315::DEBUG::2012-08-30 13:24:08,550::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users) >Thread-2315::DEBUG::2012-08-30 13:24:08,550::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it. >Thread-2315::DEBUG::2012-08-30 13:24:08,551::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records. >Thread-2315::DEBUG::2012-08-30 13:24:08,551::task::978::TaskManager.Task::(_decref) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::ref 0 aborting False >Thread-40::ERROR::2012-08-30 13:24:08,937::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu > state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info() > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info > if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-40::ERROR::2012-08-30 13:24:08,938::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-104::ERROR::2012-08-30 13:24:11,011::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed >Thread-2317::DEBUG::2012-08-30 13:24:11,410::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available >Thread-2317::DEBUG::2012-08-30 13:24:11,410::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available >Thread-40::ERROR::2012-08-30 13:24:13,939::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38> >Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect > statsFunction() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__ > retValue = self._function(*args, **kwargs) > File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet > netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) > File "/usr/share/vdsm/libvirtvm.py", line 494, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats > if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self) >libvirtError: internal error client socket is closed
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 852956
:
608047
| 608048 |
736326
|
741901