Attachment 608048 Details for Bug 852956 – /var/log/vdsm/vdsm.log

/var/log/vdsm/vdsm.log

vdsm.log (text/plain), 2.48 MB, created by EricLee on 2012-08-30 05:30:27 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: EricLee

Created: 2012-08-30 05:30:27 UTC

Size: 2.48 MB

patch

obsolete

>Thread-1554::DEBUG::2012-08-30 12:46:09,483::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1554::DEBUG::2012-08-30 12:46:09,484::task::588::TaskManager.Task::(_updateState) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::moving from state init -> state preparing
>Thread-1554::INFO::2012-08-30 12:46:09,484::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1554::DEBUG::2012-08-30 12:46:09,485::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`44070f95-838c-4ec2-8c06-f616231cfaa3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1554::DEBUG::2012-08-30 12:46:09,485::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1554::DEBUG::2012-08-30 12:46:09,485::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1554::DEBUG::2012-08-30 12:46:09,485::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`44070f95-838c-4ec2-8c06-f616231cfaa3`::Granted request
>Thread-1554::DEBUG::2012-08-30 12:46:09,486::task::817::TaskManager.Task::(resourceAcquired) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1554::DEBUG::2012-08-30 12:46:09,486::task::978::TaskManager.Task::(_decref) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::ref 1 aborting False
>Thread-1554::INFO::2012-08-30 12:46:09,489::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1554::DEBUG::2012-08-30 12:46:09,489::task::1172::TaskManager.Task::(prepare) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1554::DEBUG::2012-08-30 12:46:09,489::task::588::TaskManager.Task::(_updateState) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::moving from state preparing -> state finished
>Thread-1554::DEBUG::2012-08-30 12:46:09,490::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1554::DEBUG::2012-08-30 12:46:09,490::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1554::DEBUG::2012-08-30 12:46:09,490::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1554::DEBUG::2012-08-30 12:46:09,490::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1554::DEBUG::2012-08-30 12:46:09,491::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1554::DEBUG::2012-08-30 12:46:09,491::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1554::DEBUG::2012-08-30 12:46:09,491::task::978::TaskManager.Task::(_decref) Task=`95981bb2-59c6-464e-bf62-cc56fd80d295`::ref 0 aborting False
>Thread-1555::DEBUG::2012-08-30 12:46:13,899::task::588::TaskManager.Task::(_updateState) Task=`9b887239-83c1-4d50-8567-339e0e2b4027`::moving from state init -> state preparing
>Thread-1555::INFO::2012-08-30 12:46:13,900::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1555::INFO::2012-08-30 12:46:13,900::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164699554443', 'lastCheck': 1346301969.2454381, 'code': 0, 'valid': True}}
>Thread-1555::DEBUG::2012-08-30 12:46:13,900::task::1172::TaskManager.Task::(prepare) Task=`9b887239-83c1-4d50-8567-339e0e2b4027`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164699554443', 'lastCheck': 1346301969.2454381, 'code': 0, 'valid': True}}
>Thread-1555::DEBUG::2012-08-30 12:46:13,900::task::588::TaskManager.Task::(_updateState) Task=`9b887239-83c1-4d50-8567-339e0e2b4027`::moving from state preparing -> state finished
>Thread-1555::DEBUG::2012-08-30 12:46:13,901::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1555::DEBUG::2012-08-30 12:46:13,901::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1555::DEBUG::2012-08-30 12:46:13,901::task::978::TaskManager.Task::(_decref) Task=`9b887239-83c1-4d50-8567-339e0e2b4027`::ref 0 aborting False
>Thread-1556::DEBUG::2012-08-30 12:46:17,099::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1556::DEBUG::2012-08-30 12:46:17,100::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1558::DEBUG::2012-08-30 12:46:22,473::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1558::DEBUG::2012-08-30 12:46:22,473::task::588::TaskManager.Task::(_updateState) Task=`19d98c51-c2d9-4c41-9b5a-baaa85689617`::moving from state init -> state preparing
>Thread-1558::INFO::2012-08-30 12:46:22,474::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1558::INFO::2012-08-30 12:46:22,474::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1558::DEBUG::2012-08-30 12:46:22,475::task::1172::TaskManager.Task::(prepare) Task=`19d98c51-c2d9-4c41-9b5a-baaa85689617`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1558::DEBUG::2012-08-30 12:46:22,475::task::588::TaskManager.Task::(_updateState) Task=`19d98c51-c2d9-4c41-9b5a-baaa85689617`::moving from state preparing -> state finished
>Thread-1558::DEBUG::2012-08-30 12:46:22,475::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1558::DEBUG::2012-08-30 12:46:22,475::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1558::DEBUG::2012-08-30 12:46:22,476::task::978::TaskManager.Task::(_decref) Task=`19d98c51-c2d9-4c41-9b5a-baaa85689617`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:46:24,825::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1559::DEBUG::2012-08-30 12:46:25,693::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1559::DEBUG::2012-08-30 12:46:25,694::task::588::TaskManager.Task::(_updateState) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::moving from state init -> state preparing
>Thread-1559::INFO::2012-08-30 12:46:25,694::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1559::DEBUG::2012-08-30 12:46:25,694::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`22bbf4fe-3219-4df4-aebb-b7067f524023`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1559::DEBUG::2012-08-30 12:46:25,694::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1559::DEBUG::2012-08-30 12:46:25,695::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1559::DEBUG::2012-08-30 12:46:25,695::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`22bbf4fe-3219-4df4-aebb-b7067f524023`::Granted request
>Thread-1559::DEBUG::2012-08-30 12:46:25,695::task::817::TaskManager.Task::(resourceAcquired) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1559::DEBUG::2012-08-30 12:46:25,696::task::978::TaskManager.Task::(_decref) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::ref 1 aborting False
>Thread-1559::INFO::2012-08-30 12:46:25,698::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1559::DEBUG::2012-08-30 12:46:25,699::task::1172::TaskManager.Task::(prepare) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1559::DEBUG::2012-08-30 12:46:25,699::task::588::TaskManager.Task::(_updateState) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::moving from state preparing -> state finished
>Thread-1559::DEBUG::2012-08-30 12:46:25,699::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1559::DEBUG::2012-08-30 12:46:25,699::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1559::DEBUG::2012-08-30 12:46:25,700::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1559::DEBUG::2012-08-30 12:46:25,700::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1559::DEBUG::2012-08-30 12:46:25,700::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1559::DEBUG::2012-08-30 12:46:25,700::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1559::DEBUG::2012-08-30 12:46:25,701::task::978::TaskManager.Task::(_decref) Task=`3963f4cc-cf73-43d8-a353-1e9d6482a984`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 12:46:29,971::task::588::TaskManager.Task::(_updateState) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:46:29,971::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:46:29,972::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`979971e2-984c-46ec-8a2f-814e2a814966`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:46:29,972::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:46:29,972::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:46:29,973::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`979971e2-984c-46ec-8a2f-814e2a814966`::Granted request
>Thread-104::DEBUG::2012-08-30 12:46:29,973::task::817::TaskManager.Task::(resourceAcquired) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:46:29,973::task::978::TaskManager.Task::(_decref) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:46:29,975::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:46:29,977::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:46:29,978::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:46:29,978::task::1172::TaskManager.Task::(prepare) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:46:29,978::task::588::TaskManager.Task::(_updateState) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:46:29,979::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:46:29,979::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:46:29,979::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:46:29,979::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:46:29,980::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:46:29,980::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:46:29,980::task::978::TaskManager.Task::(_decref) Task=`34d1873f-fabc-4317-9e30-fd8d4176126a`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:46:31,832::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-104::ERROR::2012-08-30 12:46:34,994::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:46:37,953::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:46:37,954::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1563::DEBUG::2012-08-30 12:46:38,909::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1563::DEBUG::2012-08-30 12:46:38,910::task::588::TaskManager.Task::(_updateState) Task=`065c9c0c-1d8c-4558-8b13-44b3b53ffe7b`::moving from state init -> state preparing
>Thread-1563::INFO::2012-08-30 12:46:38,910::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1563::INFO::2012-08-30 12:46:38,910::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1563::DEBUG::2012-08-30 12:46:38,911::task::1172::TaskManager.Task::(prepare) Task=`065c9c0c-1d8c-4558-8b13-44b3b53ffe7b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1563::DEBUG::2012-08-30 12:46:38,911::task::588::TaskManager.Task::(_updateState) Task=`065c9c0c-1d8c-4558-8b13-44b3b53ffe7b`::moving from state preparing -> state finished
>Thread-1563::DEBUG::2012-08-30 12:46:38,911::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1563::DEBUG::2012-08-30 12:46:38,911::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1563::DEBUG::2012-08-30 12:46:38,911::task::978::TaskManager.Task::(_decref) Task=`065c9c0c-1d8c-4558-8b13-44b3b53ffe7b`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:46:40,001::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1564::DEBUG::2012-08-30 12:46:42,093::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1565::DEBUG::2012-08-30 12:46:42,093::task::588::TaskManager.Task::(_updateState) Task=`7fd8834c-7b37-4d76-b049-362778b088f1`::moving from state init -> state preparing
>Thread-1564::DEBUG::2012-08-30 12:46:42,094::task::588::TaskManager.Task::(_updateState) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::moving from state init -> state preparing
>Thread-1565::INFO::2012-08-30 12:46:42,094::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1564::INFO::2012-08-30 12:46:42,094::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1565::INFO::2012-08-30 12:46:42,094::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00248503684998', 'lastCheck': 1346301999.2711041, 'code': 0, 'valid': True}}
>Thread-1564::DEBUG::2012-08-30 12:46:42,095::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9bab8c2a-220b-45d7-baea-96407cb7c7b0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1565::DEBUG::2012-08-30 12:46:42,095::task::1172::TaskManager.Task::(prepare) Task=`7fd8834c-7b37-4d76-b049-362778b088f1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00248503684998', 'lastCheck': 1346301999.2711041, 'code': 0, 'valid': True}}
>Thread-1564::DEBUG::2012-08-30 12:46:42,095::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1565::DEBUG::2012-08-30 12:46:42,095::task::588::TaskManager.Task::(_updateState) Task=`7fd8834c-7b37-4d76-b049-362778b088f1`::moving from state preparing -> state finished
>Thread-1565::DEBUG::2012-08-30 12:46:42,096::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1565::DEBUG::2012-08-30 12:46:42,097::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1564::DEBUG::2012-08-30 12:46:42,096::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1565::DEBUG::2012-08-30 12:46:42,097::task::978::TaskManager.Task::(_decref) Task=`7fd8834c-7b37-4d76-b049-362778b088f1`::ref 0 aborting False
>Thread-1564::DEBUG::2012-08-30 12:46:42,097::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9bab8c2a-220b-45d7-baea-96407cb7c7b0`::Granted request
>Thread-1564::DEBUG::2012-08-30 12:46:42,098::task::817::TaskManager.Task::(resourceAcquired) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1564::DEBUG::2012-08-30 12:46:42,098::task::978::TaskManager.Task::(_decref) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::ref 1 aborting False
>Thread-1564::INFO::2012-08-30 12:46:42,103::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1564::DEBUG::2012-08-30 12:46:42,103::task::1172::TaskManager.Task::(prepare) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1564::DEBUG::2012-08-30 12:46:42,103::task::588::TaskManager.Task::(_updateState) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::moving from state preparing -> state finished
>Thread-1564::DEBUG::2012-08-30 12:46:42,103::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1564::DEBUG::2012-08-30 12:46:42,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1564::DEBUG::2012-08-30 12:46:42,104::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1564::DEBUG::2012-08-30 12:46:42,104::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1564::DEBUG::2012-08-30 12:46:42,104::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1564::DEBUG::2012-08-30 12:46:42,105::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1564::DEBUG::2012-08-30 12:46:42,105::task::978::TaskManager.Task::(_decref) Task=`2ff5fb18-04cf-4cea-a412-cc0806d5f66c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:46:42,955::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:46:45,003::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:46:45,003::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1566::DEBUG::2012-08-30 12:46:45,285::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1566::DEBUG::2012-08-30 12:46:45,285::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 12:46:47,956::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:46:50,004::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 12:46:52,957::task::588::TaskManager.Task::(_updateState) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:46:52,957::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:46:52,958::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ad068212-b93b-4243-a101-734b4e83c5f3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:46:52,958::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:46:52,958::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:46:52,958::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ad068212-b93b-4243-a101-734b4e83c5f3`::Granted request
>Thread-40::DEBUG::2012-08-30 12:46:52,959::task::817::TaskManager.Task::(resourceAcquired) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:46:52,959::task::978::TaskManager.Task::(_decref) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:46:52,961::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:46:52,963::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:46:52,964::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:46:52,964::task::1172::TaskManager.Task::(prepare) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:46:52,964::task::588::TaskManager.Task::(_updateState) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:46:52,965::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:46:52,965::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:46:52,965::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:46:52,965::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:46:52,966::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:46:52,966::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:46:52,966::task::978::TaskManager.Task::(_decref) Task=`a592d33d-9d2d-49cb-894b-e5ad4d3df6af`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:46:52,966::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:46:52,967::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:46:52,967::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:46:52,968::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:46:54,851::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:46:55,006::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1568::DEBUG::2012-08-30 12:46:57,318::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1568::DEBUG::2012-08-30 12:46:57,319::task::588::TaskManager.Task::(_updateState) Task=`305bf8b3-a80f-4ce6-aaae-1788d2b00479`::moving from state init -> state preparing
>Thread-1568::INFO::2012-08-30 12:46:57,319::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1568::INFO::2012-08-30 12:46:57,320::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1568::DEBUG::2012-08-30 12:46:57,320::task::1172::TaskManager.Task::(prepare) Task=`305bf8b3-a80f-4ce6-aaae-1788d2b00479`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1568::DEBUG::2012-08-30 12:46:57,320::task::588::TaskManager.Task::(_updateState) Task=`305bf8b3-a80f-4ce6-aaae-1788d2b00479`::moving from state preparing -> state finished
>Thread-1568::DEBUG::2012-08-30 12:46:57,320::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1568::DEBUG::2012-08-30 12:46:57,321::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1568::DEBUG::2012-08-30 12:46:57,321::task::978::TaskManager.Task::(_decref) Task=`305bf8b3-a80f-4ce6-aaae-1788d2b00479`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:46:57,969::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:00,007::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:00,008::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1570::DEBUG::2012-08-30 12:47:00,511::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1570::DEBUG::2012-08-30 12:47:00,511::task::588::TaskManager.Task::(_updateState) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::moving from state init -> state preparing
>Thread-1570::INFO::2012-08-30 12:47:00,512::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1570::DEBUG::2012-08-30 12:47:00,512::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e5189cb4-6c0f-4b7c-9937-61ae66d7e448`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1570::DEBUG::2012-08-30 12:47:00,512::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1570::DEBUG::2012-08-30 12:47:00,513::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1570::DEBUG::2012-08-30 12:47:00,513::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e5189cb4-6c0f-4b7c-9937-61ae66d7e448`::Granted request
>Thread-1570::DEBUG::2012-08-30 12:47:00,513::task::817::TaskManager.Task::(resourceAcquired) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1570::DEBUG::2012-08-30 12:47:00,513::task::978::TaskManager.Task::(_decref) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::ref 1 aborting False
>Thread-1570::INFO::2012-08-30 12:47:00,516::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1570::DEBUG::2012-08-30 12:47:00,516::task::1172::TaskManager.Task::(prepare) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1570::DEBUG::2012-08-30 12:47:00,517::task::588::TaskManager.Task::(_updateState) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::moving from state preparing -> state finished
>Thread-1570::DEBUG::2012-08-30 12:47:00,517::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1570::DEBUG::2012-08-30 12:47:00,517::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1570::DEBUG::2012-08-30 12:47:00,517::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1570::DEBUG::2012-08-30 12:47:00,518::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1570::DEBUG::2012-08-30 12:47:00,518::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1570::DEBUG::2012-08-30 12:47:00,518::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1570::DEBUG::2012-08-30 12:47:00,518::task::978::TaskManager.Task::(_decref) Task=`238e5093-e331-4fd0-8abb-dd56e442a097`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:47:01,858::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:47:02,970::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:05,009::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:07,971::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:07,971::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:10,010::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1573::DEBUG::2012-08-30 12:47:12,157::task::588::TaskManager.Task::(_updateState) Task=`2847ae63-91e1-4438-b6be-141f0c9f25a3`::moving from state init -> state preparing
>Thread-1573::INFO::2012-08-30 12:47:12,157::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1573::INFO::2012-08-30 12:47:12,157::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162315368652', 'lastCheck': 1346302029.291816, 'code': 0, 'valid': True}}
>Thread-1573::DEBUG::2012-08-30 12:47:12,158::task::1172::TaskManager.Task::(prepare) Task=`2847ae63-91e1-4438-b6be-141f0c9f25a3`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162315368652', 'lastCheck': 1346302029.291816, 'code': 0, 'valid': True}}
>Thread-1573::DEBUG::2012-08-30 12:47:12,158::task::588::TaskManager.Task::(_updateState) Task=`2847ae63-91e1-4438-b6be-141f0c9f25a3`::moving from state preparing -> state finished
>Thread-1573::DEBUG::2012-08-30 12:47:12,158::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1573::DEBUG::2012-08-30 12:47:12,158::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1573::DEBUG::2012-08-30 12:47:12,159::task::978::TaskManager.Task::(_decref) Task=`2847ae63-91e1-4438-b6be-141f0c9f25a3`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:47:12,972::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1574::DEBUG::2012-08-30 12:47:13,737::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1574::DEBUG::2012-08-30 12:47:13,738::task::588::TaskManager.Task::(_updateState) Task=`b0e70bef-48dc-401d-950a-d3878e592ccb`::moving from state init -> state preparing
>Thread-1574::INFO::2012-08-30 12:47:13,738::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1574::INFO::2012-08-30 12:47:13,738::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1574::DEBUG::2012-08-30 12:47:13,738::task::1172::TaskManager.Task::(prepare) Task=`b0e70bef-48dc-401d-950a-d3878e592ccb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1574::DEBUG::2012-08-30 12:47:13,739::task::588::TaskManager.Task::(_updateState) Task=`b0e70bef-48dc-401d-950a-d3878e592ccb`::moving from state preparing -> state finished
>Thread-1574::DEBUG::2012-08-30 12:47:13,739::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1574::DEBUG::2012-08-30 12:47:13,739::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1574::DEBUG::2012-08-30 12:47:13,739::task::978::TaskManager.Task::(_decref) Task=`b0e70bef-48dc-401d-950a-d3878e592ccb`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:47:15,012::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:15,012::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1575::DEBUG::2012-08-30 12:47:15,323::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1575::DEBUG::2012-08-30 12:47:15,323::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1576::DEBUG::2012-08-30 12:47:16,920::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1576::DEBUG::2012-08-30 12:47:16,920::task::588::TaskManager.Task::(_updateState) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::moving from state init -> state preparing
>Thread-1576::INFO::2012-08-30 12:47:16,920::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1576::DEBUG::2012-08-30 12:47:16,921::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ee28c1d3-325e-4180-a0b2-0fb3b7c7e6ee`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1576::DEBUG::2012-08-30 12:47:16,921::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1576::DEBUG::2012-08-30 12:47:16,921::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1576::DEBUG::2012-08-30 12:47:16,921::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ee28c1d3-325e-4180-a0b2-0fb3b7c7e6ee`::Granted request
>Thread-1576::DEBUG::2012-08-30 12:47:16,922::task::817::TaskManager.Task::(resourceAcquired) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1576::DEBUG::2012-08-30 12:47:16,922::task::978::TaskManager.Task::(_decref) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::ref 1 aborting False
>Thread-1576::INFO::2012-08-30 12:47:16,925::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1576::DEBUG::2012-08-30 12:47:16,925::task::1172::TaskManager.Task::(prepare) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1576::DEBUG::2012-08-30 12:47:16,925::task::588::TaskManager.Task::(_updateState) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::moving from state preparing -> state finished
>Thread-1576::DEBUG::2012-08-30 12:47:16,926::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1576::DEBUG::2012-08-30 12:47:16,926::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1576::DEBUG::2012-08-30 12:47:16,926::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1576::DEBUG::2012-08-30 12:47:16,926::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1576::DEBUG::2012-08-30 12:47:16,927::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1576::DEBUG::2012-08-30 12:47:16,927::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1576::DEBUG::2012-08-30 12:47:16,927::task::978::TaskManager.Task::(_decref) Task=`24d0a65b-9e9a-4781-9f90-875798456be2`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:47:17,973::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:20,013::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:22,974::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:22,974::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:47:24,875::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:47:25,015::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:27,975::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1579::DEBUG::2012-08-30 12:47:29,497::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1579::DEBUG::2012-08-30 12:47:29,498::task::588::TaskManager.Task::(_updateState) Task=`75dc896c-93c3-421b-8fc5-e88a7d08874f`::moving from state init -> state preparing
>Thread-1579::INFO::2012-08-30 12:47:29,498::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1579::INFO::2012-08-30 12:47:29,498::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1579::DEBUG::2012-08-30 12:47:29,499::task::1172::TaskManager.Task::(prepare) Task=`75dc896c-93c3-421b-8fc5-e88a7d08874f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1579::DEBUG::2012-08-30 12:47:29,499::task::588::TaskManager.Task::(_updateState) Task=`75dc896c-93c3-421b-8fc5-e88a7d08874f`::moving from state preparing -> state finished
>Thread-1579::DEBUG::2012-08-30 12:47:29,499::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1579::DEBUG::2012-08-30 12:47:29,499::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1579::DEBUG::2012-08-30 12:47:29,500::task::978::TaskManager.Task::(_decref) Task=`75dc896c-93c3-421b-8fc5-e88a7d08874f`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 12:47:30,016::task::588::TaskManager.Task::(_updateState) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:47:30,016::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:47:30,017::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a5c9b71c-250d-479c-af49-13dab4e8b200`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:47:30,017::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:47:30,017::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:47:30,017::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a5c9b71c-250d-479c-af49-13dab4e8b200`::Granted request
>Thread-104::DEBUG::2012-08-30 12:47:30,018::task::817::TaskManager.Task::(resourceAcquired) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:47:30,018::task::978::TaskManager.Task::(_decref) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:47:30,020::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:47:30,022::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:47:30,023::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:47:30,023::task::1172::TaskManager.Task::(prepare) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:47:30,023::task::588::TaskManager.Task::(_updateState) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:47:30,023::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:47:30,024::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:47:30,024::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:47:30,024::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:47:30,024::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:47:30,025::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:47:30,025::task::978::TaskManager.Task::(_decref) Task=`7f669201-712c-4d32-94df-f18ca1f450a6`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:47:30,025::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:30,026::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:30,026::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:30,027::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:47:31,882::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1581::DEBUG::2012-08-30 12:47:32,703::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1581::DEBUG::2012-08-30 12:47:32,704::task::588::TaskManager.Task::(_updateState) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::moving from state init -> state preparing
>Thread-1581::INFO::2012-08-30 12:47:32,704::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1581::DEBUG::2012-08-30 12:47:32,704::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6ea65cbb-aff7-432f-af13-22f56d8f0355`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1581::DEBUG::2012-08-30 12:47:32,705::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1581::DEBUG::2012-08-30 12:47:32,705::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1581::DEBUG::2012-08-30 12:47:32,705::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6ea65cbb-aff7-432f-af13-22f56d8f0355`::Granted request
>Thread-1581::DEBUG::2012-08-30 12:47:32,705::task::817::TaskManager.Task::(resourceAcquired) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1581::DEBUG::2012-08-30 12:47:32,706::task::978::TaskManager.Task::(_decref) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::ref 1 aborting False
>Thread-1581::INFO::2012-08-30 12:47:32,709::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1581::DEBUG::2012-08-30 12:47:32,709::task::1172::TaskManager.Task::(prepare) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1581::DEBUG::2012-08-30 12:47:32,709::task::588::TaskManager.Task::(_updateState) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::moving from state preparing -> state finished
>Thread-1581::DEBUG::2012-08-30 12:47:32,709::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1581::DEBUG::2012-08-30 12:47:32,710::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1581::DEBUG::2012-08-30 12:47:32,711::task::978::TaskManager.Task::(_decref) Task=`81c28065-958a-446b-9847-aaa2de0d527a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:47:32,976::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:35,028::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:37,977::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:37,978::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:40,029::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1583::DEBUG::2012-08-30 12:47:41,501::task::588::TaskManager.Task::(_updateState) Task=`ace23e47-08a9-4b25-8a57-a9e00c014c28`::moving from state init -> state preparing
>Thread-1583::INFO::2012-08-30 12:47:41,501::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1583::INFO::2012-08-30 12:47:41,501::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161981582642', 'lastCheck': 1346302059.3110631, 'code': 0, 'valid': True}}
>Thread-1583::DEBUG::2012-08-30 12:47:41,502::task::1172::TaskManager.Task::(prepare) Task=`ace23e47-08a9-4b25-8a57-a9e00c014c28`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161981582642', 'lastCheck': 1346302059.3110631, 'code': 0, 'valid': True}}
>Thread-1583::DEBUG::2012-08-30 12:47:41,502::task::588::TaskManager.Task::(_updateState) Task=`ace23e47-08a9-4b25-8a57-a9e00c014c28`::moving from state preparing -> state finished
>Thread-1583::DEBUG::2012-08-30 12:47:41,502::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1583::DEBUG::2012-08-30 12:47:41,502::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1583::DEBUG::2012-08-30 12:47:41,503::task::978::TaskManager.Task::(_decref) Task=`ace23e47-08a9-4b25-8a57-a9e00c014c28`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:47:42,979::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1584::DEBUG::2012-08-30 12:47:44,351::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1584::DEBUG::2012-08-30 12:47:44,352::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:47:45,031::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:47:45,031::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1585::DEBUG::2012-08-30 12:47:45,941::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1585::DEBUG::2012-08-30 12:47:45,941::task::588::TaskManager.Task::(_updateState) Task=`a565f7e6-7803-4e6b-97c2-1993312cd86f`::moving from state init -> state preparing
>Thread-1585::INFO::2012-08-30 12:47:45,941::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1585::INFO::2012-08-30 12:47:45,942::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1585::DEBUG::2012-08-30 12:47:45,942::task::1172::TaskManager.Task::(prepare) Task=`a565f7e6-7803-4e6b-97c2-1993312cd86f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1585::DEBUG::2012-08-30 12:47:45,942::task::588::TaskManager.Task::(_updateState) Task=`a565f7e6-7803-4e6b-97c2-1993312cd86f`::moving from state preparing -> state finished
>Thread-1585::DEBUG::2012-08-30 12:47:45,942::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1585::DEBUG::2012-08-30 12:47:45,943::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1585::DEBUG::2012-08-30 12:47:45,943::task::978::TaskManager.Task::(_decref) Task=`a565f7e6-7803-4e6b-97c2-1993312cd86f`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:47:47,980::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1586::DEBUG::2012-08-30 12:47:49,115::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1586::DEBUG::2012-08-30 12:47:49,116::task::588::TaskManager.Task::(_updateState) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::moving from state init -> state preparing
>Thread-1586::INFO::2012-08-30 12:47:49,116::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1586::DEBUG::2012-08-30 12:47:49,117::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`73053b8f-aade-4b88-b589-2f41888056fb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1586::DEBUG::2012-08-30 12:47:49,117::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1586::DEBUG::2012-08-30 12:47:49,117::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1586::DEBUG::2012-08-30 12:47:49,117::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`73053b8f-aade-4b88-b589-2f41888056fb`::Granted request
>Thread-1586::DEBUG::2012-08-30 12:47:49,118::task::817::TaskManager.Task::(resourceAcquired) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1586::DEBUG::2012-08-30 12:47:49,118::task::978::TaskManager.Task::(_decref) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::ref 1 aborting False
>Thread-1586::INFO::2012-08-30 12:47:49,121::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1586::DEBUG::2012-08-30 12:47:49,121::task::1172::TaskManager.Task::(prepare) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1586::DEBUG::2012-08-30 12:47:49,121::task::588::TaskManager.Task::(_updateState) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::moving from state preparing -> state finished
>Thread-1586::DEBUG::2012-08-30 12:47:49,122::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1586::DEBUG::2012-08-30 12:47:49,122::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1586::DEBUG::2012-08-30 12:47:49,122::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1586::DEBUG::2012-08-30 12:47:49,122::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1586::DEBUG::2012-08-30 12:47:49,123::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1586::DEBUG::2012-08-30 12:47:49,123::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1586::DEBUG::2012-08-30 12:47:49,123::task::978::TaskManager.Task::(_decref) Task=`9d01ff7c-08bc-454e-a7ab-03ad3e5d41a3`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:47:50,033::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 12:47:52,981::task::588::TaskManager.Task::(_updateState) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:47:52,981::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:47:52,981::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`11e3767a-4fb5-4612-b113-a5a412899247`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:47:52,981::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:47:52,982::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:47:52,982::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`11e3767a-4fb5-4612-b113-a5a412899247`::Granted request
>Thread-40::DEBUG::2012-08-30 12:47:52,982::task::817::TaskManager.Task::(resourceAcquired) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:47:52,983::task::978::TaskManager.Task::(_decref) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:47:52,984::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:47:52,987::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:47:52,988::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:47:52,988::task::1172::TaskManager.Task::(prepare) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:47:52,988::task::588::TaskManager.Task::(_updateState) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:47:52,988::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:47:52,989::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:47:52,990::task::978::TaskManager.Task::(_decref) Task=`e06f0b65-42db-4b87-ab85-84af5c98de0a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:47:52,990::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:52,991::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:52,991::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:52,991::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:47:54,905::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:47:55,034::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:47:57,992::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:00,035::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:00,036::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:48:01,911::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1590::DEBUG::2012-08-30 12:48:02,360::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1590::DEBUG::2012-08-30 12:48:02,360::task::588::TaskManager.Task::(_updateState) Task=`8774e142-89b6-42e4-81fa-1c3eb05e8a8c`::moving from state init -> state preparing
>Thread-1590::INFO::2012-08-30 12:48:02,360::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1590::INFO::2012-08-30 12:48:02,361::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1590::DEBUG::2012-08-30 12:48:02,361::task::1172::TaskManager.Task::(prepare) Task=`8774e142-89b6-42e4-81fa-1c3eb05e8a8c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1590::DEBUG::2012-08-30 12:48:02,361::task::588::TaskManager.Task::(_updateState) Task=`8774e142-89b6-42e4-81fa-1c3eb05e8a8c`::moving from state preparing -> state finished
>Thread-1590::DEBUG::2012-08-30 12:48:02,361::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1590::DEBUG::2012-08-30 12:48:02,362::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1590::DEBUG::2012-08-30 12:48:02,362::task::978::TaskManager.Task::(_decref) Task=`8774e142-89b6-42e4-81fa-1c3eb05e8a8c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:48:02,993::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:05,037::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1592::DEBUG::2012-08-30 12:48:05,564::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1592::DEBUG::2012-08-30 12:48:05,565::task::588::TaskManager.Task::(_updateState) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::moving from state init -> state preparing
>Thread-1592::INFO::2012-08-30 12:48:05,565::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1592::DEBUG::2012-08-30 12:48:05,565::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`cafb36f9-b0c9-448b-80c4-614fe06163ba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1592::DEBUG::2012-08-30 12:48:05,566::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1592::DEBUG::2012-08-30 12:48:05,566::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1592::DEBUG::2012-08-30 12:48:05,566::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`cafb36f9-b0c9-448b-80c4-614fe06163ba`::Granted request
>Thread-1592::DEBUG::2012-08-30 12:48:05,566::task::817::TaskManager.Task::(resourceAcquired) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1592::DEBUG::2012-08-30 12:48:05,567::task::978::TaskManager.Task::(_decref) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::ref 1 aborting False
>Thread-1592::INFO::2012-08-30 12:48:05,570::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1592::DEBUG::2012-08-30 12:48:05,570::task::1172::TaskManager.Task::(prepare) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1592::DEBUG::2012-08-30 12:48:05,570::task::588::TaskManager.Task::(_updateState) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::moving from state preparing -> state finished
>Thread-1592::DEBUG::2012-08-30 12:48:05,570::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1592::DEBUG::2012-08-30 12:48:05,571::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1592::DEBUG::2012-08-30 12:48:05,572::task::978::TaskManager.Task::(_decref) Task=`b5d39bcf-7874-4322-ac83-ca5a9b1b46f3`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:48:07,994::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:07,995::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1593::DEBUG::2012-08-30 12:48:09,939::task::588::TaskManager.Task::(_updateState) Task=`053dd472-8fc9-4b69-a5cb-489330f04201`::moving from state init -> state preparing
>Thread-1593::INFO::2012-08-30 12:48:09,939::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1593::INFO::2012-08-30 12:48:09,939::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346302089.3315029, 'code': 0, 'valid': True}}
>Thread-1593::DEBUG::2012-08-30 12:48:09,940::task::1172::TaskManager.Task::(prepare) Task=`053dd472-8fc9-4b69-a5cb-489330f04201`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346302089.3315029, 'code': 0, 'valid': True}}
>Thread-1593::DEBUG::2012-08-30 12:48:09,940::task::588::TaskManager.Task::(_updateState) Task=`053dd472-8fc9-4b69-a5cb-489330f04201`::moving from state preparing -> state finished
>Thread-1593::DEBUG::2012-08-30 12:48:09,940::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1593::DEBUG::2012-08-30 12:48:09,940::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1593::DEBUG::2012-08-30 12:48:09,940::task::978::TaskManager.Task::(_decref) Task=`053dd472-8fc9-4b69-a5cb-489330f04201`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:48:10,038::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:12,995::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1594::DEBUG::2012-08-30 12:48:13,347::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1594::DEBUG::2012-08-30 12:48:13,347::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:48:15,039::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:15,040::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:17,996::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1596::DEBUG::2012-08-30 12:48:18,623::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1596::DEBUG::2012-08-30 12:48:18,624::task::588::TaskManager.Task::(_updateState) Task=`f858818b-d1e8-4c95-a47f-36dbc40f7286`::moving from state init -> state preparing
>Thread-1596::INFO::2012-08-30 12:48:18,624::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1596::INFO::2012-08-30 12:48:18,625::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1596::DEBUG::2012-08-30 12:48:18,625::task::1172::TaskManager.Task::(prepare) Task=`f858818b-d1e8-4c95-a47f-36dbc40f7286`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1596::DEBUG::2012-08-30 12:48:18,625::task::588::TaskManager.Task::(_updateState) Task=`f858818b-d1e8-4c95-a47f-36dbc40f7286`::moving from state preparing -> state finished
>Thread-1596::DEBUG::2012-08-30 12:48:18,625::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1596::DEBUG::2012-08-30 12:48:18,626::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1596::DEBUG::2012-08-30 12:48:18,626::task::978::TaskManager.Task::(_decref) Task=`f858818b-d1e8-4c95-a47f-36dbc40f7286`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:48:20,041::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1597::DEBUG::2012-08-30 12:48:21,828::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1597::DEBUG::2012-08-30 12:48:21,829::task::588::TaskManager.Task::(_updateState) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::moving from state init -> state preparing
>Thread-1597::INFO::2012-08-30 12:48:21,829::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1597::DEBUG::2012-08-30 12:48:21,830::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8c87deda-5c0f-4208-bb61-1d66f9a9ccc1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1597::DEBUG::2012-08-30 12:48:21,830::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1597::DEBUG::2012-08-30 12:48:21,830::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1597::DEBUG::2012-08-30 12:48:21,830::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8c87deda-5c0f-4208-bb61-1d66f9a9ccc1`::Granted request
>Thread-1597::DEBUG::2012-08-30 12:48:21,831::task::817::TaskManager.Task::(resourceAcquired) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1597::DEBUG::2012-08-30 12:48:21,831::task::978::TaskManager.Task::(_decref) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::ref 1 aborting False
>Thread-1597::INFO::2012-08-30 12:48:21,834::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1597::DEBUG::2012-08-30 12:48:21,834::task::1172::TaskManager.Task::(prepare) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1597::DEBUG::2012-08-30 12:48:21,834::task::588::TaskManager.Task::(_updateState) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::moving from state preparing -> state finished
>Thread-1597::DEBUG::2012-08-30 12:48:21,835::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1597::DEBUG::2012-08-30 12:48:21,835::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1597::DEBUG::2012-08-30 12:48:21,835::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1597::DEBUG::2012-08-30 12:48:21,835::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1597::DEBUG::2012-08-30 12:48:21,836::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1597::DEBUG::2012-08-30 12:48:21,836::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1597::DEBUG::2012-08-30 12:48:21,836::task::978::TaskManager.Task::(_decref) Task=`6a491836-90f6-4b01-85ad-0c3003effe76`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:48:22,997::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:22,998::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:48:24,933::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:48:25,042::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:27,999::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:48:30,043::task::588::TaskManager.Task::(_updateState) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:48:30,044::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:48:30,044::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`476a2dae-0fc4-4390-ba9b-7f0309c2b10b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:48:30,044::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:48:30,045::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:48:30,045::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`476a2dae-0fc4-4390-ba9b-7f0309c2b10b`::Granted request
>Thread-104::DEBUG::2012-08-30 12:48:30,045::task::817::TaskManager.Task::(resourceAcquired) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:48:30,045::task::978::TaskManager.Task::(_decref) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:48:30,047::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:48:30,049::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:48:30,050::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:48:30,050::task::1172::TaskManager.Task::(prepare) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:48:30,051::task::588::TaskManager.Task::(_updateState) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:48:30,051::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:48:30,051::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:48:30,051::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:48:30,052::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:48:30,052::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:48:30,052::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:48:30,052::task::978::TaskManager.Task::(_decref) Task=`bc77198d-be47-4539-b149-66e6c980ec8d`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:48:30,053::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:30,053::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:30,053::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:30,054::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:48:31,940::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:48:33,000::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:35,055::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1601::DEBUG::2012-08-30 12:48:35,082::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1601::DEBUG::2012-08-30 12:48:35,083::task::588::TaskManager.Task::(_updateState) Task=`e7d6243e-039a-4ea8-abea-69594a335359`::moving from state init -> state preparing
>Thread-1601::INFO::2012-08-30 12:48:35,083::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1601::INFO::2012-08-30 12:48:35,083::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1601::DEBUG::2012-08-30 12:48:35,084::task::1172::TaskManager.Task::(prepare) Task=`e7d6243e-039a-4ea8-abea-69594a335359`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1601::DEBUG::2012-08-30 12:48:35,084::task::588::TaskManager.Task::(_updateState) Task=`e7d6243e-039a-4ea8-abea-69594a335359`::moving from state preparing -> state finished
>Thread-1601::DEBUG::2012-08-30 12:48:35,084::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1601::DEBUG::2012-08-30 12:48:35,084::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1601::DEBUG::2012-08-30 12:48:35,085::task::978::TaskManager.Task::(_decref) Task=`e7d6243e-039a-4ea8-abea-69594a335359`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:48:38,000::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:38,001::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1602::DEBUG::2012-08-30 12:48:38,265::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1603::DEBUG::2012-08-30 12:48:38,265::task::588::TaskManager.Task::(_updateState) Task=`96fe9764-0b7f-409a-9ae7-f763e05c1be8`::moving from state init -> state preparing
>Thread-1602::DEBUG::2012-08-30 12:48:38,265::task::588::TaskManager.Task::(_updateState) Task=`a410bb23-2bce-4806-88e9-40875c336007`::moving from state init -> state preparing
>Thread-1603::INFO::2012-08-30 12:48:38,266::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1602::INFO::2012-08-30 12:48:38,266::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1603::INFO::2012-08-30 12:48:38,266::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163698196411', 'lastCheck': 1346302109.3441539, 'code': 0, 'valid': True}}
>Thread-1602::DEBUG::2012-08-30 12:48:38,266::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`7676593c-682b-48c3-9ff2-9122225f2b21`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1603::DEBUG::2012-08-30 12:48:38,267::task::1172::TaskManager.Task::(prepare) Task=`96fe9764-0b7f-409a-9ae7-f763e05c1be8`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163698196411', 'lastCheck': 1346302109.3441539, 'code': 0, 'valid': True}}
>Thread-1602::DEBUG::2012-08-30 12:48:38,267::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1603::DEBUG::2012-08-30 12:48:38,267::task::588::TaskManager.Task::(_updateState) Task=`96fe9764-0b7f-409a-9ae7-f763e05c1be8`::moving from state preparing -> state finished
>Thread-1602::DEBUG::2012-08-30 12:48:38,267::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1603::DEBUG::2012-08-30 12:48:38,267::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1602::DEBUG::2012-08-30 12:48:38,268::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`7676593c-682b-48c3-9ff2-9122225f2b21`::Granted request
>Thread-1603::DEBUG::2012-08-30 12:48:38,268::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1602::DEBUG::2012-08-30 12:48:38,268::task::817::TaskManager.Task::(resourceAcquired) Task=`a410bb23-2bce-4806-88e9-40875c336007`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1603::DEBUG::2012-08-30 12:48:38,268::task::978::TaskManager.Task::(_decref) Task=`96fe9764-0b7f-409a-9ae7-f763e05c1be8`::ref 0 aborting False
>Thread-1602::DEBUG::2012-08-30 12:48:38,269::task::978::TaskManager.Task::(_decref) Task=`a410bb23-2bce-4806-88e9-40875c336007`::ref 1 aborting False
>Thread-1602::INFO::2012-08-30 12:48:38,273::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1602::DEBUG::2012-08-30 12:48:38,274::task::1172::TaskManager.Task::(prepare) Task=`a410bb23-2bce-4806-88e9-40875c336007`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1602::DEBUG::2012-08-30 12:48:38,274::task::588::TaskManager.Task::(_updateState) Task=`a410bb23-2bce-4806-88e9-40875c336007`::moving from state preparing -> state finished
>Thread-1602::DEBUG::2012-08-30 12:48:38,274::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1602::DEBUG::2012-08-30 12:48:38,274::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1602::DEBUG::2012-08-30 12:48:38,275::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1602::DEBUG::2012-08-30 12:48:38,275::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1602::DEBUG::2012-08-30 12:48:38,275::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1602::DEBUG::2012-08-30 12:48:38,275::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1602::DEBUG::2012-08-30 12:48:38,276::task::978::TaskManager.Task::(_decref) Task=`a410bb23-2bce-4806-88e9-40875c336007`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:48:40,056::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1604::DEBUG::2012-08-30 12:48:41,473::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1604::DEBUG::2012-08-30 12:48:41,474::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 12:48:43,002::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:45,057::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:45,058::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:48,003::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:48:50,059::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1606::DEBUG::2012-08-30 12:48:51,643::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1606::DEBUG::2012-08-30 12:48:51,644::task::588::TaskManager.Task::(_updateState) Task=`d676892e-e340-4076-94b4-ad5321e9df5c`::moving from state init -> state preparing
>Thread-1606::INFO::2012-08-30 12:48:51,644::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1606::INFO::2012-08-30 12:48:51,644::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1606::DEBUG::2012-08-30 12:48:51,645::task::1172::TaskManager.Task::(prepare) Task=`d676892e-e340-4076-94b4-ad5321e9df5c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1606::DEBUG::2012-08-30 12:48:51,645::task::588::TaskManager.Task::(_updateState) Task=`d676892e-e340-4076-94b4-ad5321e9df5c`::moving from state preparing -> state finished
>Thread-1606::DEBUG::2012-08-30 12:48:51,645::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1606::DEBUG::2012-08-30 12:48:51,645::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1606::DEBUG::2012-08-30 12:48:51,646::task::978::TaskManager.Task::(_decref) Task=`d676892e-e340-4076-94b4-ad5321e9df5c`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 12:48:53,004::task::588::TaskManager.Task::(_updateState) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:48:53,004::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:48:53,004::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`599a8e2c-2172-4941-8252-75cdcfc0e440`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:48:53,005::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:48:53,005::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:48:53,005::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`599a8e2c-2172-4941-8252-75cdcfc0e440`::Granted request
>Thread-40::DEBUG::2012-08-30 12:48:53,006::task::817::TaskManager.Task::(resourceAcquired) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:48:53,006::task::978::TaskManager.Task::(_decref) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:48:53,008::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:48:53,010::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:48:53,011::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:48:53,011::task::1172::TaskManager.Task::(prepare) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:48:53,011::task::588::TaskManager.Task::(_updateState) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:48:53,012::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:48:53,012::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:48:53,012::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:48:53,012::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:48:53,013::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:48:53,013::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:48:53,013::task::978::TaskManager.Task::(_decref) Task=`a3fcabba-a0f2-495a-9834-e2d870329637`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:48:53,013::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:53,014::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:53,014::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:53,015::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1608::DEBUG::2012-08-30 12:48:54,815::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1608::DEBUG::2012-08-30 12:48:54,815::task::588::TaskManager.Task::(_updateState) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::moving from state init -> state preparing
>Thread-1608::INFO::2012-08-30 12:48:54,816::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1608::DEBUG::2012-08-30 12:48:54,816::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8b29fa7b-a030-4f43-ad1d-de2b9c0e7da4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1608::DEBUG::2012-08-30 12:48:54,816::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1608::DEBUG::2012-08-30 12:48:54,817::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1608::DEBUG::2012-08-30 12:48:54,817::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8b29fa7b-a030-4f43-ad1d-de2b9c0e7da4`::Granted request
>Thread-1608::DEBUG::2012-08-30 12:48:54,817::task::817::TaskManager.Task::(resourceAcquired) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1608::DEBUG::2012-08-30 12:48:54,817::task::978::TaskManager.Task::(_decref) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::ref 1 aborting False
>Thread-1608::INFO::2012-08-30 12:48:54,820::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1608::DEBUG::2012-08-30 12:48:54,821::task::1172::TaskManager.Task::(prepare) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1608::DEBUG::2012-08-30 12:48:54,821::task::588::TaskManager.Task::(_updateState) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::moving from state preparing -> state finished
>Thread-1608::DEBUG::2012-08-30 12:48:54,821::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1608::DEBUG::2012-08-30 12:48:54,821::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1608::DEBUG::2012-08-30 12:48:54,822::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1608::DEBUG::2012-08-30 12:48:54,822::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1608::DEBUG::2012-08-30 12:48:54,822::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1608::DEBUG::2012-08-30 12:48:54,822::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1608::DEBUG::2012-08-30 12:48:54,822::task::978::TaskManager.Task::(_decref) Task=`eb2f9500-02db-4702-9b36-6f9ffdecc3c9`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:48:54,958::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:48:55,060::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:48:58,016::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:00,062::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:00,062::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:49:01,966::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:49:03,017::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:05,063::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1611::DEBUG::2012-08-30 12:49:06,463::task::588::TaskManager.Task::(_updateState) Task=`73c4895c-4ea6-449a-af4b-cbe4ff42fea7`::moving from state init -> state preparing
>Thread-1611::INFO::2012-08-30 12:49:06,464::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1611::INFO::2012-08-30 12:49:06,464::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178503990173', 'lastCheck': 1346302139.36462, 'code': 0, 'valid': True}}
>Thread-1611::DEBUG::2012-08-30 12:49:06,464::task::1172::TaskManager.Task::(prepare) Task=`73c4895c-4ea6-449a-af4b-cbe4ff42fea7`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178503990173', 'lastCheck': 1346302139.36462, 'code': 0, 'valid': True}}
>Thread-1611::DEBUG::2012-08-30 12:49:06,464::task::588::TaskManager.Task::(_updateState) Task=`73c4895c-4ea6-449a-af4b-cbe4ff42fea7`::moving from state preparing -> state finished
>Thread-1611::DEBUG::2012-08-30 12:49:06,464::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1611::DEBUG::2012-08-30 12:49:06,465::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1611::DEBUG::2012-08-30 12:49:06,465::task::978::TaskManager.Task::(_decref) Task=`73c4895c-4ea6-449a-af4b-cbe4ff42fea7`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:49:08,018::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:08,018::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1612::DEBUG::2012-08-30 12:49:08,054::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1612::DEBUG::2012-08-30 12:49:08,054::task::588::TaskManager.Task::(_updateState) Task=`ecccd816-c460-47d9-80b3-84f57e64b169`::moving from state init -> state preparing
>Thread-1612::INFO::2012-08-30 12:49:08,055::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1612::INFO::2012-08-30 12:49:08,055::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1612::DEBUG::2012-08-30 12:49:08,055::task::1172::TaskManager.Task::(prepare) Task=`ecccd816-c460-47d9-80b3-84f57e64b169`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1612::DEBUG::2012-08-30 12:49:08,055::task::588::TaskManager.Task::(_updateState) Task=`ecccd816-c460-47d9-80b3-84f57e64b169`::moving from state preparing -> state finished
>Thread-1612::DEBUG::2012-08-30 12:49:08,056::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1612::DEBUG::2012-08-30 12:49:08,056::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1612::DEBUG::2012-08-30 12:49:08,056::task::978::TaskManager.Task::(_decref) Task=`ecccd816-c460-47d9-80b3-84f57e64b169`::ref 0 aborting False
>Thread-1613::DEBUG::2012-08-30 12:49:09,938::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1613::DEBUG::2012-08-30 12:49:09,938::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:49:10,064::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1614::DEBUG::2012-08-30 12:49:11,530::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1614::DEBUG::2012-08-30 12:49:11,531::task::588::TaskManager.Task::(_updateState) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::moving from state init -> state preparing
>Thread-1614::INFO::2012-08-30 12:49:11,531::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1614::DEBUG::2012-08-30 12:49:11,531::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5fa49e40-e9a7-4b83-b3a5-91140095bce8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1614::DEBUG::2012-08-30 12:49:11,532::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1614::DEBUG::2012-08-30 12:49:11,532::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1614::DEBUG::2012-08-30 12:49:11,532::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5fa49e40-e9a7-4b83-b3a5-91140095bce8`::Granted request
>Thread-1614::DEBUG::2012-08-30 12:49:11,532::task::817::TaskManager.Task::(resourceAcquired) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1614::DEBUG::2012-08-30 12:49:11,533::task::978::TaskManager.Task::(_decref) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::ref 1 aborting False
>Thread-1614::INFO::2012-08-30 12:49:11,536::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1614::DEBUG::2012-08-30 12:49:11,536::task::1172::TaskManager.Task::(prepare) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1614::DEBUG::2012-08-30 12:49:11,536::task::588::TaskManager.Task::(_updateState) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::moving from state preparing -> state finished
>Thread-1614::DEBUG::2012-08-30 12:49:11,536::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1614::DEBUG::2012-08-30 12:49:11,537::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1614::DEBUG::2012-08-30 12:49:11,538::task::978::TaskManager.Task::(_decref) Task=`fad6d364-0e3f-4835-a016-dcd28e47dd1e`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:49:13,019::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:15,066::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:15,066::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:18,020::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:20,067::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:23,021::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:23,021::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1617::DEBUG::2012-08-30 12:49:24,734::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1617::DEBUG::2012-08-30 12:49:24,735::task::588::TaskManager.Task::(_updateState) Task=`77695686-65ff-4d7e-ac8c-9c839784809a`::moving from state init -> state preparing
>Thread-1617::INFO::2012-08-30 12:49:24,735::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1617::INFO::2012-08-30 12:49:24,736::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1617::DEBUG::2012-08-30 12:49:24,736::task::1172::TaskManager.Task::(prepare) Task=`77695686-65ff-4d7e-ac8c-9c839784809a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1617::DEBUG::2012-08-30 12:49:24,736::task::588::TaskManager.Task::(_updateState) Task=`77695686-65ff-4d7e-ac8c-9c839784809a`::moving from state preparing -> state finished
>Thread-1617::DEBUG::2012-08-30 12:49:24,736::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1617::DEBUG::2012-08-30 12:49:24,736::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1617::DEBUG::2012-08-30 12:49:24,737::task::978::TaskManager.Task::(_decref) Task=`77695686-65ff-4d7e-ac8c-9c839784809a`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:49:24,985::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:49:25,069::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1619::DEBUG::2012-08-30 12:49:27,916::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1619::DEBUG::2012-08-30 12:49:27,917::task::588::TaskManager.Task::(_updateState) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::moving from state init -> state preparing
>Thread-1619::INFO::2012-08-30 12:49:27,917::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1619::DEBUG::2012-08-30 12:49:27,917::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`54593cdb-ddb3-4fc0-9e6f-aea678e3a1a4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1619::DEBUG::2012-08-30 12:49:27,918::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1619::DEBUG::2012-08-30 12:49:27,918::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1619::DEBUG::2012-08-30 12:49:27,918::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`54593cdb-ddb3-4fc0-9e6f-aea678e3a1a4`::Granted request
>Thread-1619::DEBUG::2012-08-30 12:49:27,919::task::817::TaskManager.Task::(resourceAcquired) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1619::DEBUG::2012-08-30 12:49:27,919::task::978::TaskManager.Task::(_decref) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::ref 1 aborting False
>Thread-1619::INFO::2012-08-30 12:49:27,922::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1619::DEBUG::2012-08-30 12:49:27,922::task::1172::TaskManager.Task::(prepare) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1619::DEBUG::2012-08-30 12:49:27,922::task::588::TaskManager.Task::(_updateState) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::moving from state preparing -> state finished
>Thread-1619::DEBUG::2012-08-30 12:49:27,922::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1619::DEBUG::2012-08-30 12:49:27,923::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1619::DEBUG::2012-08-30 12:49:27,923::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1619::DEBUG::2012-08-30 12:49:27,923::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1619::DEBUG::2012-08-30 12:49:27,923::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1619::DEBUG::2012-08-30 12:49:27,924::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1619::DEBUG::2012-08-30 12:49:27,924::task::978::TaskManager.Task::(_decref) Task=`32f642e1-fcbd-4c03-a1d1-bbdf9de471f9`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:49:28,022::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:49:30,070::task::588::TaskManager.Task::(_updateState) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:49:30,070::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:49:30,070::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`550455b7-4232-4de6-8504-514c1ccd9f2a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:49:30,071::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:49:30,071::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:49:30,071::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`550455b7-4232-4de6-8504-514c1ccd9f2a`::Granted request
>Thread-104::DEBUG::2012-08-30 12:49:30,071::task::817::TaskManager.Task::(resourceAcquired) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:49:30,072::task::978::TaskManager.Task::(_decref) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:49:30,073::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:49:30,075::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:49:30,076::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:49:30,077::task::1172::TaskManager.Task::(prepare) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:49:30,077::task::588::TaskManager.Task::(_updateState) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:49:30,077::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:49:30,077::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:49:30,078::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:49:30,078::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:49:30,078::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:49:30,078::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:49:30,078::task::978::TaskManager.Task::(_decref) Task=`12c2f7f6-a721-4c06-b61b-507ac912396c`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:49:30,079::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:30,079::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:30,080::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:30,080::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:49:31,990::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:49:33,023::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:35,081::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:38,024::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:38,024::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1621::DEBUG::2012-08-30 12:49:38,069::task::588::TaskManager.Task::(_updateState) Task=`2e7cf250-552b-41ce-9af7-d5c2730adfa0`::moving from state init -> state preparing
>Thread-1621::INFO::2012-08-30 12:49:38,069::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1621::INFO::2012-08-30 12:49:38,069::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163388252258', 'lastCheck': 1346302169.3848281, 'code': 0, 'valid': True}}
>Thread-1621::DEBUG::2012-08-30 12:49:38,070::task::1172::TaskManager.Task::(prepare) Task=`2e7cf250-552b-41ce-9af7-d5c2730adfa0`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163388252258', 'lastCheck': 1346302169.3848281, 'code': 0, 'valid': True}}
>Thread-1621::DEBUG::2012-08-30 12:49:38,070::task::588::TaskManager.Task::(_updateState) Task=`2e7cf250-552b-41ce-9af7-d5c2730adfa0`::moving from state preparing -> state finished
>Thread-1621::DEBUG::2012-08-30 12:49:38,070::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1621::DEBUG::2012-08-30 12:49:38,070::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1621::DEBUG::2012-08-30 12:49:38,071::task::978::TaskManager.Task::(_decref) Task=`2e7cf250-552b-41ce-9af7-d5c2730adfa0`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:49:40,082::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1622::DEBUG::2012-08-30 12:49:42,335::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1623::DEBUG::2012-08-30 12:49:42,335::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1622::DEBUG::2012-08-30 12:49:42,336::task::588::TaskManager.Task::(_updateState) Task=`10145ac5-3c4e-4c29-989b-892c00af43b8`::moving from state init -> state preparing
>Thread-1623::DEBUG::2012-08-30 12:49:42,336::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1622::INFO::2012-08-30 12:49:42,336::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1622::INFO::2012-08-30 12:49:42,337::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1622::DEBUG::2012-08-30 12:49:42,338::task::1172::TaskManager.Task::(prepare) Task=`10145ac5-3c4e-4c29-989b-892c00af43b8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1622::DEBUG::2012-08-30 12:49:42,338::task::588::TaskManager.Task::(_updateState) Task=`10145ac5-3c4e-4c29-989b-892c00af43b8`::moving from state preparing -> state finished
>Thread-1622::DEBUG::2012-08-30 12:49:42,338::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1622::DEBUG::2012-08-30 12:49:42,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1622::DEBUG::2012-08-30 12:49:42,339::task::978::TaskManager.Task::(_decref) Task=`10145ac5-3c4e-4c29-989b-892c00af43b8`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:49:43,025::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:45,084::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:45,084::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1624::DEBUG::2012-08-30 12:49:45,996::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1624::DEBUG::2012-08-30 12:49:45,996::task::588::TaskManager.Task::(_updateState) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::moving from state init -> state preparing
>Thread-1624::INFO::2012-08-30 12:49:45,996::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1624::DEBUG::2012-08-30 12:49:45,997::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0524ec4b-e8ac-4e3a-be65-793a4fc37f5e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1624::DEBUG::2012-08-30 12:49:45,997::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1624::DEBUG::2012-08-30 12:49:45,997::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1624::DEBUG::2012-08-30 12:49:45,997::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0524ec4b-e8ac-4e3a-be65-793a4fc37f5e`::Granted request
>Thread-1624::DEBUG::2012-08-30 12:49:45,998::task::817::TaskManager.Task::(resourceAcquired) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1624::DEBUG::2012-08-30 12:49:45,998::task::978::TaskManager.Task::(_decref) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::ref 1 aborting False
>Thread-1624::INFO::2012-08-30 12:49:46,001::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1624::DEBUG::2012-08-30 12:49:46,001::task::1172::TaskManager.Task::(prepare) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1624::DEBUG::2012-08-30 12:49:46,001::task::588::TaskManager.Task::(_updateState) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::moving from state preparing -> state finished
>Thread-1624::DEBUG::2012-08-30 12:49:46,002::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1624::DEBUG::2012-08-30 12:49:46,002::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1624::DEBUG::2012-08-30 12:49:46,002::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1624::DEBUG::2012-08-30 12:49:46,002::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1624::DEBUG::2012-08-30 12:49:46,003::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1624::DEBUG::2012-08-30 12:49:46,003::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1624::DEBUG::2012-08-30 12:49:46,003::task::978::TaskManager.Task::(_decref) Task=`7bc7a8e8-1ee5-436f-8a50-1b1be5c3242a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:49:48,026::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:49:50,085::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 12:49:53,027::task::588::TaskManager.Task::(_updateState) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:49:53,028::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:49:53,028::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c919ef2a-8971-4660-8335-eb74302d190b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:49:53,028::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:49:53,028::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:49:53,029::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c919ef2a-8971-4660-8335-eb74302d190b`::Granted request
>Thread-40::DEBUG::2012-08-30 12:49:53,029::task::817::TaskManager.Task::(resourceAcquired) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:49:53,029::task::978::TaskManager.Task::(_decref) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:49:53,031::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:49:53,033::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:49:53,034::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:49:53,034::task::1172::TaskManager.Task::(prepare) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:49:53,035::task::588::TaskManager.Task::(_updateState) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:49:53,035::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:49:53,035::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:49:53,035::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:49:53,036::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:49:53,036::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:49:53,036::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:49:53,036::task::978::TaskManager.Task::(_decref) Task=`00596be6-813c-42ac-ba06-13e0fe2e01ba`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:49:53,037::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:53,037::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:53,038::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:53,038::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:49:55,011::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:49:55,087::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:49:58,039::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1628::DEBUG::2012-08-30 12:49:59,214::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1628::DEBUG::2012-08-30 12:49:59,214::task::588::TaskManager.Task::(_updateState) Task=`0510cd18-91e1-4306-8417-c7ffaaa5475a`::moving from state init -> state preparing
>Thread-1628::INFO::2012-08-30 12:49:59,214::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1628::INFO::2012-08-30 12:49:59,215::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1628::DEBUG::2012-08-30 12:49:59,215::task::1172::TaskManager.Task::(prepare) Task=`0510cd18-91e1-4306-8417-c7ffaaa5475a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1628::DEBUG::2012-08-30 12:49:59,215::task::588::TaskManager.Task::(_updateState) Task=`0510cd18-91e1-4306-8417-c7ffaaa5475a`::moving from state preparing -> state finished
>Thread-1628::DEBUG::2012-08-30 12:49:59,215::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1628::DEBUG::2012-08-30 12:49:59,215::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1628::DEBUG::2012-08-30 12:49:59,216::task::978::TaskManager.Task::(_decref) Task=`0510cd18-91e1-4306-8417-c7ffaaa5475a`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:50:00,088::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:00,088::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:50:02,018::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1629::DEBUG::2012-08-30 12:50:02,402::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1629::DEBUG::2012-08-30 12:50:02,402::task::588::TaskManager.Task::(_updateState) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::moving from state init -> state preparing
>Thread-1629::INFO::2012-08-30 12:50:02,402::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1629::DEBUG::2012-08-30 12:50:02,403::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`de30bf54-f9c0-4fb8-8b1c-a7b9b4a7fc0f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1629::DEBUG::2012-08-30 12:50:02,403::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1629::DEBUG::2012-08-30 12:50:02,404::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1629::DEBUG::2012-08-30 12:50:02,404::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`de30bf54-f9c0-4fb8-8b1c-a7b9b4a7fc0f`::Granted request
>Thread-1629::DEBUG::2012-08-30 12:50:02,405::task::817::TaskManager.Task::(resourceAcquired) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1629::DEBUG::2012-08-30 12:50:02,405::task::978::TaskManager.Task::(_decref) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::ref 1 aborting False
>Thread-1629::INFO::2012-08-30 12:50:02,408::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1629::DEBUG::2012-08-30 12:50:02,408::task::1172::TaskManager.Task::(prepare) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1629::DEBUG::2012-08-30 12:50:02,408::task::588::TaskManager.Task::(_updateState) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::moving from state preparing -> state finished
>Thread-1629::DEBUG::2012-08-30 12:50:02,409::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1629::DEBUG::2012-08-30 12:50:02,409::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1629::DEBUG::2012-08-30 12:50:02,409::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1629::DEBUG::2012-08-30 12:50:02,409::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1629::DEBUG::2012-08-30 12:50:02,410::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1629::DEBUG::2012-08-30 12:50:02,410::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1629::DEBUG::2012-08-30 12:50:02,410::task::978::TaskManager.Task::(_decref) Task=`15795d61-bb74-4eac-a17b-d2f1dcaa9f51`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:50:03,040::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:05,089::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1631::DEBUG::2012-08-30 12:50:07,584::task::588::TaskManager.Task::(_updateState) Task=`beb78f1e-de0e-435e-8a29-623ea8d18ec1`::moving from state init -> state preparing
>Thread-1631::INFO::2012-08-30 12:50:07,584::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1631::INFO::2012-08-30 12:50:07,585::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177097320557', 'lastCheck': 1346302199.4052939, 'code': 0, 'valid': True}}
>Thread-1631::DEBUG::2012-08-30 12:50:07,585::task::1172::TaskManager.Task::(prepare) Task=`beb78f1e-de0e-435e-8a29-623ea8d18ec1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177097320557', 'lastCheck': 1346302199.4052939, 'code': 0, 'valid': True}}
>Thread-1631::DEBUG::2012-08-30 12:50:07,585::task::588::TaskManager.Task::(_updateState) Task=`beb78f1e-de0e-435e-8a29-623ea8d18ec1`::moving from state preparing -> state finished
>Thread-1631::DEBUG::2012-08-30 12:50:07,585::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1631::DEBUG::2012-08-30 12:50:07,586::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1631::DEBUG::2012-08-30 12:50:07,586::task::978::TaskManager.Task::(_decref) Task=`beb78f1e-de0e-435e-8a29-623ea8d18ec1`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:50:08,041::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:08,041::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:10,090::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1632::DEBUG::2012-08-30 12:50:10,764::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1632::DEBUG::2012-08-30 12:50:10,764::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 12:50:13,042::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:15,091::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:15,092::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1633::DEBUG::2012-08-30 12:50:15,719::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1633::DEBUG::2012-08-30 12:50:15,719::task::588::TaskManager.Task::(_updateState) Task=`6015da28-ed4f-4f81-a716-cd8a60257d79`::moving from state init -> state preparing
>Thread-1633::INFO::2012-08-30 12:50:15,719::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1633::INFO::2012-08-30 12:50:15,720::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1633::DEBUG::2012-08-30 12:50:15,720::task::1172::TaskManager.Task::(prepare) Task=`6015da28-ed4f-4f81-a716-cd8a60257d79`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1633::DEBUG::2012-08-30 12:50:15,720::task::588::TaskManager.Task::(_updateState) Task=`6015da28-ed4f-4f81-a716-cd8a60257d79`::moving from state preparing -> state finished
>Thread-1633::DEBUG::2012-08-30 12:50:15,720::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1633::DEBUG::2012-08-30 12:50:15,721::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1633::DEBUG::2012-08-30 12:50:15,721::task::978::TaskManager.Task::(_decref) Task=`6015da28-ed4f-4f81-a716-cd8a60257d79`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:50:18,044::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1635::DEBUG::2012-08-30 12:50:18,918::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1635::DEBUG::2012-08-30 12:50:18,918::task::588::TaskManager.Task::(_updateState) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::moving from state init -> state preparing
>Thread-1635::INFO::2012-08-30 12:50:18,919::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1635::DEBUG::2012-08-30 12:50:18,919::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1f1070f9-a5f0-41d2-93b2-860551c01d07`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1635::DEBUG::2012-08-30 12:50:18,919::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1635::DEBUG::2012-08-30 12:50:18,920::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1635::DEBUG::2012-08-30 12:50:18,920::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1f1070f9-a5f0-41d2-93b2-860551c01d07`::Granted request
>Thread-1635::DEBUG::2012-08-30 12:50:18,920::task::817::TaskManager.Task::(resourceAcquired) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1635::DEBUG::2012-08-30 12:50:18,920::task::978::TaskManager.Task::(_decref) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::ref 1 aborting False
>Thread-1635::INFO::2012-08-30 12:50:18,923::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1635::DEBUG::2012-08-30 12:50:18,924::task::1172::TaskManager.Task::(prepare) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1635::DEBUG::2012-08-30 12:50:18,924::task::588::TaskManager.Task::(_updateState) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::moving from state preparing -> state finished
>Thread-1635::DEBUG::2012-08-30 12:50:18,924::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1635::DEBUG::2012-08-30 12:50:18,924::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1635::DEBUG::2012-08-30 12:50:18,925::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1635::DEBUG::2012-08-30 12:50:18,925::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1635::DEBUG::2012-08-30 12:50:18,925::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1635::DEBUG::2012-08-30 12:50:18,925::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1635::DEBUG::2012-08-30 12:50:18,926::task::978::TaskManager.Task::(_decref) Task=`17b06efc-2080-4963-ae2c-43489a2630fe`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:50:20,092::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:23,045::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:23,046::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:50:25,039::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:50:25,093::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:28,047::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-26::DEBUG::2012-08-30 12:50:29,418::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 12:50:29,418::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-26::DEBUG::2012-08-30 12:50:29,419::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None)
>Thread-26::DEBUG::2012-08-30 12:50:29,553::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = '  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5
>Thread-26::WARNING::2012-08-30 12:50:29,554::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found']
>Thread-26::DEBUG::2012-08-30 12:50:29,554::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-26::DEBUG::2012-08-30 12:50:29,559::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 12:50:29,560::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend
>Thread-26::DEBUG::2012-08-30 12:50:29,562::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d']
>Thread-26::DEBUG::2012-08-30 12:50:29,563::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-26::WARNING::2012-08-30 12:50:29,564::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered
>Thread-26::WARNING::2012-08-30 12:50:29,564::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered
>Thread-104::DEBUG::2012-08-30 12:50:30,094::task::588::TaskManager.Task::(_updateState) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:50:30,095::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:50:30,095::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ac0ac618-5b41-43a0-82a4-43005baab9c4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:50:30,095::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:50:30,096::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:50:30,096::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ac0ac618-5b41-43a0-82a4-43005baab9c4`::Granted request
>Thread-104::DEBUG::2012-08-30 12:50:30,096::task::817::TaskManager.Task::(resourceAcquired) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:50:30,097::task::978::TaskManager.Task::(_decref) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:50:30,098::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:50:30,100::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:50:30,101::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:50:30,102::task::1172::TaskManager.Task::(prepare) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:50:30,102::task::588::TaskManager.Task::(_updateState) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:50:30,102::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:50:30,102::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:50:30,103::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:50:30,103::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:50:30,103::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:50:30,103::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:50:30,104::task::978::TaskManager.Task::(_decref) Task=`6c9eb371-1603-423b-9143-beba7aafff78`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:50:30,104::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:30,104::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:30,105::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:30,105::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:50:32,046::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1639::DEBUG::2012-08-30 12:50:32,124::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1639::DEBUG::2012-08-30 12:50:32,125::task::588::TaskManager.Task::(_updateState) Task=`6887233b-e5d6-49b5-98cd-5a94956d479e`::moving from state init -> state preparing
>Thread-1639::INFO::2012-08-30 12:50:32,125::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1639::INFO::2012-08-30 12:50:32,125::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1639::DEBUG::2012-08-30 12:50:32,125::task::1172::TaskManager.Task::(prepare) Task=`6887233b-e5d6-49b5-98cd-5a94956d479e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1639::DEBUG::2012-08-30 12:50:32,126::task::588::TaskManager.Task::(_updateState) Task=`6887233b-e5d6-49b5-98cd-5a94956d479e`::moving from state preparing -> state finished
>Thread-1639::DEBUG::2012-08-30 12:50:32,126::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1639::DEBUG::2012-08-30 12:50:32,126::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1639::DEBUG::2012-08-30 12:50:32,126::task::978::TaskManager.Task::(_decref) Task=`6887233b-e5d6-49b5-98cd-5a94956d479e`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:50:33,048::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:35,106::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1645::DEBUG::2012-08-30 12:50:35,303::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1646::DEBUG::2012-08-30 12:50:35,304::task::588::TaskManager.Task::(_updateState) Task=`aab85601-58d4-431f-97c1-75afac3e4b13`::moving from state init -> state preparing
>Thread-1645::DEBUG::2012-08-30 12:50:35,304::task::588::TaskManager.Task::(_updateState) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::moving from state init -> state preparing
>Thread-1646::INFO::2012-08-30 12:50:35,305::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1645::INFO::2012-08-30 12:50:35,305::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1646::INFO::2012-08-30 12:50:35,305::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00158500671387', 'lastCheck': 1346302229.5707181, 'code': 0, 'valid': True}}
>Thread-1645::DEBUG::2012-08-30 12:50:35,305::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`83062314-1940-4b2f-8dfa-f8d8be9a6d7e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1646::DEBUG::2012-08-30 12:50:35,306::task::1172::TaskManager.Task::(prepare) Task=`aab85601-58d4-431f-97c1-75afac3e4b13`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00158500671387', 'lastCheck': 1346302229.5707181, 'code': 0, 'valid': True}}
>Thread-1645::DEBUG::2012-08-30 12:50:35,306::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1646::DEBUG::2012-08-30 12:50:35,306::task::588::TaskManager.Task::(_updateState) Task=`aab85601-58d4-431f-97c1-75afac3e4b13`::moving from state preparing -> state finished
>Thread-1645::DEBUG::2012-08-30 12:50:35,306::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1646::DEBUG::2012-08-30 12:50:35,306::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1645::DEBUG::2012-08-30 12:50:35,307::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`83062314-1940-4b2f-8dfa-f8d8be9a6d7e`::Granted request
>Thread-1646::DEBUG::2012-08-30 12:50:35,307::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1645::DEBUG::2012-08-30 12:50:35,307::task::817::TaskManager.Task::(resourceAcquired) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1646::DEBUG::2012-08-30 12:50:35,307::task::978::TaskManager.Task::(_decref) Task=`aab85601-58d4-431f-97c1-75afac3e4b13`::ref 0 aborting False
>Thread-1645::DEBUG::2012-08-30 12:50:35,308::task::978::TaskManager.Task::(_decref) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::ref 1 aborting False
>Thread-1645::INFO::2012-08-30 12:50:35,312::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1645::DEBUG::2012-08-30 12:50:35,313::task::1172::TaskManager.Task::(prepare) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1645::DEBUG::2012-08-30 12:50:35,313::task::588::TaskManager.Task::(_updateState) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::moving from state preparing -> state finished
>Thread-1645::DEBUG::2012-08-30 12:50:35,313::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1645::DEBUG::2012-08-30 12:50:35,313::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1645::DEBUG::2012-08-30 12:50:35,314::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1645::DEBUG::2012-08-30 12:50:35,314::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1645::DEBUG::2012-08-30 12:50:35,314::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1645::DEBUG::2012-08-30 12:50:35,314::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1645::DEBUG::2012-08-30 12:50:35,315::task::978::TaskManager.Task::(_decref) Task=`63d6ad4c-fe59-433d-87b5-0766772c8c45`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:50:38,049::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:38,050::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1647::DEBUG::2012-08-30 12:50:38,483::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1647::DEBUG::2012-08-30 12:50:38,483::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:50:40,107::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:43,051::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:45,108::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:50:45,109::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:48,052::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1649::DEBUG::2012-08-30 12:50:48,582::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1649::DEBUG::2012-08-30 12:50:48,583::task::588::TaskManager.Task::(_updateState) Task=`c0b99daf-1319-4d64-9902-578266598894`::moving from state init -> state preparing
>Thread-1649::INFO::2012-08-30 12:50:48,583::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1649::INFO::2012-08-30 12:50:48,583::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1649::DEBUG::2012-08-30 12:50:48,583::task::1172::TaskManager.Task::(prepare) Task=`c0b99daf-1319-4d64-9902-578266598894`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1649::DEBUG::2012-08-30 12:50:48,584::task::588::TaskManager.Task::(_updateState) Task=`c0b99daf-1319-4d64-9902-578266598894`::moving from state preparing -> state finished
>Thread-1649::DEBUG::2012-08-30 12:50:48,584::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1649::DEBUG::2012-08-30 12:50:48,584::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1649::DEBUG::2012-08-30 12:50:48,584::task::978::TaskManager.Task::(_decref) Task=`c0b99daf-1319-4d64-9902-578266598894`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:50:50,110::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1651::DEBUG::2012-08-30 12:50:51,780::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1651::DEBUG::2012-08-30 12:50:51,781::task::588::TaskManager.Task::(_updateState) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::moving from state init -> state preparing
>Thread-1651::INFO::2012-08-30 12:50:51,781::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1651::DEBUG::2012-08-30 12:50:51,781::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1d29f26f-26f7-4b1b-8b3a-a52f5197583a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1651::DEBUG::2012-08-30 12:50:51,782::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1651::DEBUG::2012-08-30 12:50:51,782::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1651::DEBUG::2012-08-30 12:50:51,782::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1d29f26f-26f7-4b1b-8b3a-a52f5197583a`::Granted request
>Thread-1651::DEBUG::2012-08-30 12:50:51,782::task::817::TaskManager.Task::(resourceAcquired) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1651::DEBUG::2012-08-30 12:50:51,783::task::978::TaskManager.Task::(_decref) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::ref 1 aborting False
>Thread-1651::INFO::2012-08-30 12:50:51,786::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1651::DEBUG::2012-08-30 12:50:51,786::task::1172::TaskManager.Task::(prepare) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1651::DEBUG::2012-08-30 12:50:51,786::task::588::TaskManager.Task::(_updateState) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::moving from state preparing -> state finished
>Thread-1651::DEBUG::2012-08-30 12:50:51,786::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1651::DEBUG::2012-08-30 12:50:51,786::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1651::DEBUG::2012-08-30 12:50:51,787::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1651::DEBUG::2012-08-30 12:50:51,787::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1651::DEBUG::2012-08-30 12:50:51,787::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1651::DEBUG::2012-08-30 12:50:51,787::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1651::DEBUG::2012-08-30 12:50:51,788::task::978::TaskManager.Task::(_decref) Task=`fed49cdf-5cb8-4fb9-b257-4513e44a9ee5`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 12:50:53,053::task::588::TaskManager.Task::(_updateState) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:50:53,054::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:50:53,054::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`18cfb057-2388-418e-907d-3c6b75587037`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:50:53,054::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:50:53,055::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:50:53,055::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`18cfb057-2388-418e-907d-3c6b75587037`::Granted request
>Thread-40::DEBUG::2012-08-30 12:50:53,055::task::817::TaskManager.Task::(resourceAcquired) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:50:53,056::task::978::TaskManager.Task::(_decref) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:50:53,057::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:50:53,059::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:50:53,061::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:50:53,061::task::1172::TaskManager.Task::(prepare) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:50:53,061::task::588::TaskManager.Task::(_updateState) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:50:53,061::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:50:53,061::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:50:53,062::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:50:53,062::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:50:53,062::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:50:53,062::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:50:53,063::task::978::TaskManager.Task::(_decref) Task=`7adc629c-63a9-4b38-b4b7-b3e611cca98c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:50:53,063::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:53,063::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:53,064::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:53,064::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:50:55,068::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:50:55,111::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:50:58,066::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:00,111::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:00,112::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:51:02,074::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:51:03,067::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1654::DEBUG::2012-08-30 12:51:03,438::task::588::TaskManager.Task::(_updateState) Task=`38f998ee-7b4d-41d8-addb-6e41bade9906`::moving from state init -> state preparing
>Thread-1654::INFO::2012-08-30 12:51:03,438::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1654::INFO::2012-08-30 12:51:03,439::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177693367004', 'lastCheck': 1346302259.595521, 'code': 0, 'valid': True}}
>Thread-1654::DEBUG::2012-08-30 12:51:03,439::task::1172::TaskManager.Task::(prepare) Task=`38f998ee-7b4d-41d8-addb-6e41bade9906`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177693367004', 'lastCheck': 1346302259.595521, 'code': 0, 'valid': True}}
>Thread-1654::DEBUG::2012-08-30 12:51:03,439::task::588::TaskManager.Task::(_updateState) Task=`38f998ee-7b4d-41d8-addb-6e41bade9906`::moving from state preparing -> state finished
>Thread-1654::DEBUG::2012-08-30 12:51:03,439::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1654::DEBUG::2012-08-30 12:51:03,439::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1654::DEBUG::2012-08-30 12:51:03,440::task::978::TaskManager.Task::(_decref) Task=`38f998ee-7b4d-41d8-addb-6e41bade9906`::ref 0 aborting False
>Thread-1655::DEBUG::2012-08-30 12:51:05,031::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1655::DEBUG::2012-08-30 12:51:05,031::task::588::TaskManager.Task::(_updateState) Task=`41068544-9c30-4986-9b8f-bad4c6e13694`::moving from state init -> state preparing
>Thread-1655::INFO::2012-08-30 12:51:05,031::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1655::INFO::2012-08-30 12:51:05,032::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1655::DEBUG::2012-08-30 12:51:05,032::task::1172::TaskManager.Task::(prepare) Task=`41068544-9c30-4986-9b8f-bad4c6e13694`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1655::DEBUG::2012-08-30 12:51:05,032::task::588::TaskManager.Task::(_updateState) Task=`41068544-9c30-4986-9b8f-bad4c6e13694`::moving from state preparing -> state finished
>Thread-1655::DEBUG::2012-08-30 12:51:05,032::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1655::DEBUG::2012-08-30 12:51:05,033::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1655::DEBUG::2012-08-30 12:51:05,033::task::978::TaskManager.Task::(_decref) Task=`41068544-9c30-4986-9b8f-bad4c6e13694`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:51:05,113::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1656::DEBUG::2012-08-30 12:51:06,619::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1656::DEBUG::2012-08-30 12:51:06,620::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 12:51:08,068::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:08,069::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1657::DEBUG::2012-08-30 12:51:08,223::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1657::DEBUG::2012-08-30 12:51:08,223::task::588::TaskManager.Task::(_updateState) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::moving from state init -> state preparing
>Thread-1657::INFO::2012-08-30 12:51:08,224::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1657::DEBUG::2012-08-30 12:51:08,224::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8e2f79e2-5000-4d08-9616-961ca5287152`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1657::DEBUG::2012-08-30 12:51:08,224::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1657::DEBUG::2012-08-30 12:51:08,225::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1657::DEBUG::2012-08-30 12:51:08,225::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8e2f79e2-5000-4d08-9616-961ca5287152`::Granted request
>Thread-1657::DEBUG::2012-08-30 12:51:08,226::task::817::TaskManager.Task::(resourceAcquired) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1657::DEBUG::2012-08-30 12:51:08,226::task::978::TaskManager.Task::(_decref) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::ref 1 aborting False
>Thread-1657::INFO::2012-08-30 12:51:08,229::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1657::DEBUG::2012-08-30 12:51:08,229::task::1172::TaskManager.Task::(prepare) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1657::DEBUG::2012-08-30 12:51:08,229::task::588::TaskManager.Task::(_updateState) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::moving from state preparing -> state finished
>Thread-1657::DEBUG::2012-08-30 12:51:08,230::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1657::DEBUG::2012-08-30 12:51:08,230::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1657::DEBUG::2012-08-30 12:51:08,230::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1657::DEBUG::2012-08-30 12:51:08,230::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1657::DEBUG::2012-08-30 12:51:08,231::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1657::DEBUG::2012-08-30 12:51:08,231::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1657::DEBUG::2012-08-30 12:51:08,231::task::978::TaskManager.Task::(_decref) Task=`aec3538e-fa94-4486-9e08-e410cd1be7f4`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:51:10,114::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:13,070::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:15,115::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:15,115::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:18,071::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:20,116::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1660::DEBUG::2012-08-30 12:51:21,441::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1660::DEBUG::2012-08-30 12:51:21,442::task::588::TaskManager.Task::(_updateState) Task=`12661056-e037-4b0a-8f19-8aba801ff02b`::moving from state init -> state preparing
>Thread-1660::INFO::2012-08-30 12:51:21,442::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1660::INFO::2012-08-30 12:51:21,442::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1660::DEBUG::2012-08-30 12:51:21,442::task::1172::TaskManager.Task::(prepare) Task=`12661056-e037-4b0a-8f19-8aba801ff02b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1660::DEBUG::2012-08-30 12:51:21,443::task::588::TaskManager.Task::(_updateState) Task=`12661056-e037-4b0a-8f19-8aba801ff02b`::moving from state preparing -> state finished
>Thread-1660::DEBUG::2012-08-30 12:51:21,443::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1660::DEBUG::2012-08-30 12:51:21,443::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1660::DEBUG::2012-08-30 12:51:21,443::task::978::TaskManager.Task::(_decref) Task=`12661056-e037-4b0a-8f19-8aba801ff02b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:51:23,073::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:23,073::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1662::DEBUG::2012-08-30 12:51:24,625::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1662::DEBUG::2012-08-30 12:51:24,626::task::588::TaskManager.Task::(_updateState) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::moving from state init -> state preparing
>Thread-1662::INFO::2012-08-30 12:51:24,626::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1662::DEBUG::2012-08-30 12:51:24,626::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ed14358f-633e-4100-b8ad-5dbe6293f9fd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1662::DEBUG::2012-08-30 12:51:24,627::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1662::DEBUG::2012-08-30 12:51:24,627::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1662::DEBUG::2012-08-30 12:51:24,627::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ed14358f-633e-4100-b8ad-5dbe6293f9fd`::Granted request
>Thread-1662::DEBUG::2012-08-30 12:51:24,627::task::817::TaskManager.Task::(resourceAcquired) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1662::DEBUG::2012-08-30 12:51:24,628::task::978::TaskManager.Task::(_decref) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::ref 1 aborting False
>Thread-1662::INFO::2012-08-30 12:51:24,631::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1662::DEBUG::2012-08-30 12:51:24,631::task::1172::TaskManager.Task::(prepare) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1662::DEBUG::2012-08-30 12:51:24,631::task::588::TaskManager.Task::(_updateState) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::moving from state preparing -> state finished
>Thread-1662::DEBUG::2012-08-30 12:51:24,631::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1662::DEBUG::2012-08-30 12:51:24,631::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1662::DEBUG::2012-08-30 12:51:24,632::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1662::DEBUG::2012-08-30 12:51:24,632::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1662::DEBUG::2012-08-30 12:51:24,632::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1662::DEBUG::2012-08-30 12:51:24,632::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1662::DEBUG::2012-08-30 12:51:24,633::task::978::TaskManager.Task::(_decref) Task=`a5df896a-125e-4fae-80b4-df08a911b7ca`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:51:25,094::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:51:25,117::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:28,074::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:51:30,118::task::588::TaskManager.Task::(_updateState) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:51:30,119::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:51:30,119::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`7b942bf1-9ea4-421e-b399-3ad6daeca5b7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:51:30,119::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:51:30,120::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:51:30,120::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`7b942bf1-9ea4-421e-b399-3ad6daeca5b7`::Granted request
>Thread-104::DEBUG::2012-08-30 12:51:30,120::task::817::TaskManager.Task::(resourceAcquired) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:51:30,120::task::978::TaskManager.Task::(_decref) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:51:30,122::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:51:30,124::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:51:30,125::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:51:30,125::task::1172::TaskManager.Task::(prepare) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:51:30,125::task::588::TaskManager.Task::(_updateState) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:51:30,126::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:51:30,126::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:51:30,126::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:51:30,126::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:51:30,127::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:51:30,127::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:51:30,127::task::978::TaskManager.Task::(_decref) Task=`2f47f029-706b-4dcd-96ba-134fe71311bd`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:51:30,128::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:30,128::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:30,128::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:30,129::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:51:32,100::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1664::DEBUG::2012-08-30 12:51:32,735::task::588::TaskManager.Task::(_updateState) Task=`39e366dc-6ee4-4c35-9974-ea796d8c2904`::moving from state init -> state preparing
>Thread-1664::INFO::2012-08-30 12:51:32,736::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1664::INFO::2012-08-30 12:51:32,736::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346302289.6153109, 'code': 0, 'valid': True}}
>Thread-1664::DEBUG::2012-08-30 12:51:32,736::task::1172::TaskManager.Task::(prepare) Task=`39e366dc-6ee4-4c35-9974-ea796d8c2904`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346302289.6153109, 'code': 0, 'valid': True}}
>Thread-1664::DEBUG::2012-08-30 12:51:32,737::task::588::TaskManager.Task::(_updateState) Task=`39e366dc-6ee4-4c35-9974-ea796d8c2904`::moving from state preparing -> state finished
>Thread-1664::DEBUG::2012-08-30 12:51:32,737::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1664::DEBUG::2012-08-30 12:51:32,737::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1664::DEBUG::2012-08-30 12:51:32,737::task::978::TaskManager.Task::(_decref) Task=`39e366dc-6ee4-4c35-9974-ea796d8c2904`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:51:33,076::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:35,130::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1665::DEBUG::2012-08-30 12:51:36,205::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1665::DEBUG::2012-08-30 12:51:36,206::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1666::DEBUG::2012-08-30 12:51:37,797::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1666::DEBUG::2012-08-30 12:51:37,798::task::588::TaskManager.Task::(_updateState) Task=`3e8f7ca4-e61b-4dfc-8a60-5246274228a2`::moving from state init -> state preparing
>Thread-1666::INFO::2012-08-30 12:51:37,798::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1666::INFO::2012-08-30 12:51:37,798::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1666::DEBUG::2012-08-30 12:51:37,798::task::1172::TaskManager.Task::(prepare) Task=`3e8f7ca4-e61b-4dfc-8a60-5246274228a2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1666::DEBUG::2012-08-30 12:51:37,799::task::588::TaskManager.Task::(_updateState) Task=`3e8f7ca4-e61b-4dfc-8a60-5246274228a2`::moving from state preparing -> state finished
>Thread-1666::DEBUG::2012-08-30 12:51:37,799::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1666::DEBUG::2012-08-30 12:51:37,799::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1666::DEBUG::2012-08-30 12:51:37,799::task::978::TaskManager.Task::(_decref) Task=`3e8f7ca4-e61b-4dfc-8a60-5246274228a2`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:51:38,077::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:38,078::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:40,131::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1667::DEBUG::2012-08-30 12:51:41,013::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1667::DEBUG::2012-08-30 12:51:41,014::task::588::TaskManager.Task::(_updateState) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::moving from state init -> state preparing
>Thread-1667::INFO::2012-08-30 12:51:41,014::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1667::DEBUG::2012-08-30 12:51:41,015::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3a0552d9-a90a-4c12-a765-81ef7ae07c99`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1667::DEBUG::2012-08-30 12:51:41,015::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1667::DEBUG::2012-08-30 12:51:41,015::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1667::DEBUG::2012-08-30 12:51:41,016::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3a0552d9-a90a-4c12-a765-81ef7ae07c99`::Granted request
>Thread-1667::DEBUG::2012-08-30 12:51:41,016::task::817::TaskManager.Task::(resourceAcquired) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1667::DEBUG::2012-08-30 12:51:41,016::task::978::TaskManager.Task::(_decref) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::ref 1 aborting False
>Thread-1667::INFO::2012-08-30 12:51:41,019::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1667::DEBUG::2012-08-30 12:51:41,019::task::1172::TaskManager.Task::(prepare) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1667::DEBUG::2012-08-30 12:51:41,020::task::588::TaskManager.Task::(_updateState) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::moving from state preparing -> state finished
>Thread-1667::DEBUG::2012-08-30 12:51:41,020::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1667::DEBUG::2012-08-30 12:51:41,020::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1667::DEBUG::2012-08-30 12:51:41,020::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1667::DEBUG::2012-08-30 12:51:41,021::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1667::DEBUG::2012-08-30 12:51:41,021::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1667::DEBUG::2012-08-30 12:51:41,021::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1667::DEBUG::2012-08-30 12:51:41,021::task::978::TaskManager.Task::(_decref) Task=`343e52c7-d681-4b23-ae78-fffd8a0f00c6`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:51:43,079::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:45,132::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:45,132::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:48,080::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:51:50,133::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 12:51:53,081::task::588::TaskManager.Task::(_updateState) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:51:53,082::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:51:53,082::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9307a3e0-c66c-4031-a41f-78b3a4e56bf3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:51:53,082::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:51:53,083::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:51:53,083::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9307a3e0-c66c-4031-a41f-78b3a4e56bf3`::Granted request
>Thread-40::DEBUG::2012-08-30 12:51:53,083::task::817::TaskManager.Task::(resourceAcquired) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:51:53,084::task::978::TaskManager.Task::(_decref) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:51:53,086::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:51:53,088::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:51:53,090::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:51:53,090::task::1172::TaskManager.Task::(prepare) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:51:53,090::task::588::TaskManager.Task::(_updateState) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:51:53,090::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:51:53,090::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:51:53,091::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:51:53,091::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:51:53,091::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:51:53,091::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:51:53,092::task::978::TaskManager.Task::(_decref) Task=`42a0be39-8fa2-4853-b92a-2c8108fe92aa`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:51:53,092::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:53,093::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:53,093::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:51:53,093::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1671::DEBUG::2012-08-30 12:51:54,231::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1671::DEBUG::2012-08-30 12:51:54,232::task::588::TaskManager.Task::(_updateState) Task=`395563fb-8486-471f-a4f6-1dadc9446a18`::moving from state init -> state preparing
>Thread-1671::INFO::2012-08-30 12:51:54,232::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1671::INFO::2012-08-30 12:51:54,232::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1671::DEBUG::2012-08-30 12:51:54,233::task::1172::TaskManager.Task::(prepare) Task=`395563fb-8486-471f-a4f6-1dadc9446a18`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1671::DEBUG::2012-08-30 12:51:54,233::task::588::TaskManager.Task::(_updateState) Task=`395563fb-8486-471f-a4f6-1dadc9446a18`::moving from state preparing -> state finished
>Thread-1671::DEBUG::2012-08-30 12:51:54,233::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1671::DEBUG::2012-08-30 12:51:54,233::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1671::DEBUG::2012-08-30 12:51:54,234::task::978::TaskManager.Task::(_decref) Task=`395563fb-8486-471f-a4f6-1dadc9446a18`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:51:55,120::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-104::ERROR::2012-08-30 12:51:55,134::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1673::DEBUG::2012-08-30 12:51:57,441::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1673::DEBUG::2012-08-30 12:51:57,441::task::588::TaskManager.Task::(_updateState) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::moving from state init -> state preparing
>Thread-1673::INFO::2012-08-30 12:51:57,441::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1673::DEBUG::2012-08-30 12:51:57,442::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f8017968-c826-417c-b4d2-fc81cdcf2490`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1673::DEBUG::2012-08-30 12:51:57,442::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1673::DEBUG::2012-08-30 12:51:57,442::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1673::DEBUG::2012-08-30 12:51:57,443::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f8017968-c826-417c-b4d2-fc81cdcf2490`::Granted request
>Thread-1673::DEBUG::2012-08-30 12:51:57,443::task::817::TaskManager.Task::(resourceAcquired) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1673::DEBUG::2012-08-30 12:51:57,443::task::978::TaskManager.Task::(_decref) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::ref 1 aborting False
>Thread-1673::INFO::2012-08-30 12:51:57,446::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1673::DEBUG::2012-08-30 12:51:57,446::task::1172::TaskManager.Task::(prepare) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1673::DEBUG::2012-08-30 12:51:57,447::task::588::TaskManager.Task::(_updateState) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::moving from state preparing -> state finished
>Thread-1673::DEBUG::2012-08-30 12:51:57,447::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1673::DEBUG::2012-08-30 12:51:57,447::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1673::DEBUG::2012-08-30 12:51:57,447::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1673::DEBUG::2012-08-30 12:51:57,448::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1673::DEBUG::2012-08-30 12:51:57,448::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1673::DEBUG::2012-08-30 12:51:57,448::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1673::DEBUG::2012-08-30 12:51:57,448::task::978::TaskManager.Task::(_decref) Task=`c5e92b1b-00f5-40d5-a409-33f11c982e01`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:51:58,095::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:00,135::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:00,135::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1674::DEBUG::2012-08-30 12:52:01,783::task::588::TaskManager.Task::(_updateState) Task=`0ff8264a-ed9f-443c-9296-ff1323a963dd`::moving from state init -> state preparing
>Thread-1674::INFO::2012-08-30 12:52:01,783::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1674::INFO::2012-08-30 12:52:01,784::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177407264709', 'lastCheck': 1346302319.635859, 'code': 0, 'valid': True}}
>Thread-1674::DEBUG::2012-08-30 12:52:01,784::task::1172::TaskManager.Task::(prepare) Task=`0ff8264a-ed9f-443c-9296-ff1323a963dd`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177407264709', 'lastCheck': 1346302319.635859, 'code': 0, 'valid': True}}
>Thread-1674::DEBUG::2012-08-30 12:52:01,784::task::588::TaskManager.Task::(_updateState) Task=`0ff8264a-ed9f-443c-9296-ff1323a963dd`::moving from state preparing -> state finished
>Thread-1674::DEBUG::2012-08-30 12:52:01,784::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1674::DEBUG::2012-08-30 12:52:01,785::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1674::DEBUG::2012-08-30 12:52:01,785::task::978::TaskManager.Task::(_decref) Task=`0ff8264a-ed9f-443c-9296-ff1323a963dd`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:52:02,127::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:52:03,096::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1675::DEBUG::2012-08-30 12:52:05,013::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1675::DEBUG::2012-08-30 12:52:05,014::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:52:05,136::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:08,097::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:08,098::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:10,137::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1677::DEBUG::2012-08-30 12:52:10,314::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1677::DEBUG::2012-08-30 12:52:10,315::task::588::TaskManager.Task::(_updateState) Task=`185a03ad-12dd-4f38-9a58-c536b559ee29`::moving from state init -> state preparing
>Thread-1677::INFO::2012-08-30 12:52:10,315::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1677::INFO::2012-08-30 12:52:10,315::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1677::DEBUG::2012-08-30 12:52:10,316::task::1172::TaskManager.Task::(prepare) Task=`185a03ad-12dd-4f38-9a58-c536b559ee29`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1677::DEBUG::2012-08-30 12:52:10,316::task::588::TaskManager.Task::(_updateState) Task=`185a03ad-12dd-4f38-9a58-c536b559ee29`::moving from state preparing -> state finished
>Thread-1677::DEBUG::2012-08-30 12:52:10,316::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1677::DEBUG::2012-08-30 12:52:10,317::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1677::DEBUG::2012-08-30 12:52:10,317::task::978::TaskManager.Task::(_decref) Task=`185a03ad-12dd-4f38-9a58-c536b559ee29`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:52:13,099::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1678::DEBUG::2012-08-30 12:52:13,837::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1678::DEBUG::2012-08-30 12:52:13,837::task::588::TaskManager.Task::(_updateState) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::moving from state init -> state preparing
>Thread-1678::INFO::2012-08-30 12:52:13,838::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1678::DEBUG::2012-08-30 12:52:13,838::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d11ffc6e-744d-4863-95c0-265aa80a2b88`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1678::DEBUG::2012-08-30 12:52:13,838::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1678::DEBUG::2012-08-30 12:52:13,838::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1678::DEBUG::2012-08-30 12:52:13,839::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d11ffc6e-744d-4863-95c0-265aa80a2b88`::Granted request
>Thread-1678::DEBUG::2012-08-30 12:52:13,839::task::817::TaskManager.Task::(resourceAcquired) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1678::DEBUG::2012-08-30 12:52:13,839::task::978::TaskManager.Task::(_decref) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::ref 1 aborting False
>Thread-1678::INFO::2012-08-30 12:52:13,842::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1678::DEBUG::2012-08-30 12:52:13,842::task::1172::TaskManager.Task::(prepare) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1678::DEBUG::2012-08-30 12:52:13,843::task::588::TaskManager.Task::(_updateState) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::moving from state preparing -> state finished
>Thread-1678::DEBUG::2012-08-30 12:52:13,843::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1678::DEBUG::2012-08-30 12:52:13,843::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1678::DEBUG::2012-08-30 12:52:13,843::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1678::DEBUG::2012-08-30 12:52:13,844::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1678::DEBUG::2012-08-30 12:52:13,844::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1678::DEBUG::2012-08-30 12:52:13,844::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1678::DEBUG::2012-08-30 12:52:13,844::task::978::TaskManager.Task::(_decref) Task=`d933a8f9-b5e9-477c-9809-8d43e8ed2aca`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:52:15,138::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:15,139::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:18,100::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:20,140::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:23,102::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:23,102::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:25,140::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:52:25,144::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1682::DEBUG::2012-08-30 12:52:27,057::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1682::DEBUG::2012-08-30 12:52:27,058::task::588::TaskManager.Task::(_updateState) Task=`a422784f-d661-493e-bda3-217b760138ad`::moving from state init -> state preparing
>Thread-1682::INFO::2012-08-30 12:52:27,058::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1682::INFO::2012-08-30 12:52:27,058::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1682::DEBUG::2012-08-30 12:52:27,059::task::1172::TaskManager.Task::(prepare) Task=`a422784f-d661-493e-bda3-217b760138ad`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1682::DEBUG::2012-08-30 12:52:27,059::task::588::TaskManager.Task::(_updateState) Task=`a422784f-d661-493e-bda3-217b760138ad`::moving from state preparing -> state finished
>Thread-1682::DEBUG::2012-08-30 12:52:27,059::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1682::DEBUG::2012-08-30 12:52:27,059::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1682::DEBUG::2012-08-30 12:52:27,059::task::978::TaskManager.Task::(_decref) Task=`a422784f-d661-493e-bda3-217b760138ad`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:52:28,103::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:52:30,141::task::588::TaskManager.Task::(_updateState) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:52:30,142::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:52:30,142::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`94378283-af42-4b94-ba7e-8198df2ace1b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:52:30,142::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:52:30,143::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:52:30,143::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`94378283-af42-4b94-ba7e-8198df2ace1b`::Granted request
>Thread-104::DEBUG::2012-08-30 12:52:30,143::task::817::TaskManager.Task::(resourceAcquired) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:52:30,143::task::978::TaskManager.Task::(_decref) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:52:30,145::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:52:30,147::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:52:30,148::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:52:30,148::task::1172::TaskManager.Task::(prepare) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:52:30,149::task::588::TaskManager.Task::(_updateState) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:52:30,149::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:52:30,149::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:52:30,149::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:52:30,150::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:52:30,150::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:52:30,150::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:52:30,150::task::978::TaskManager.Task::(_decref) Task=`b5f548e2-6503-4482-8cc7-5d2741e901d4`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:52:30,151::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:30,151::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:30,151::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:30,152::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1683::DEBUG::2012-08-30 12:52:30,242::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1684::DEBUG::2012-08-30 12:52:30,243::task::588::TaskManager.Task::(_updateState) Task=`6804587e-fbc2-4bbf-b9ae-3ba82fdab845`::moving from state init -> state preparing
>Thread-1683::DEBUG::2012-08-30 12:52:30,243::task::588::TaskManager.Task::(_updateState) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::moving from state init -> state preparing
>Thread-1684::INFO::2012-08-30 12:52:30,243::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1683::INFO::2012-08-30 12:52:30,243::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1684::INFO::2012-08-30 12:52:30,243::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161409378052', 'lastCheck': 1346302349.654927, 'code': 0, 'valid': True}}
>Thread-1683::DEBUG::2012-08-30 12:52:30,244::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f01980df-e962-4791-814a-b3a6045de8c7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1684::DEBUG::2012-08-30 12:52:30,244::task::1172::TaskManager.Task::(prepare) Task=`6804587e-fbc2-4bbf-b9ae-3ba82fdab845`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161409378052', 'lastCheck': 1346302349.654927, 'code': 0, 'valid': True}}
>Thread-1683::DEBUG::2012-08-30 12:52:30,244::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1684::DEBUG::2012-08-30 12:52:30,244::task::588::TaskManager.Task::(_updateState) Task=`6804587e-fbc2-4bbf-b9ae-3ba82fdab845`::moving from state preparing -> state finished
>Thread-1683::DEBUG::2012-08-30 12:52:30,245::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1684::DEBUG::2012-08-30 12:52:30,245::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1683::DEBUG::2012-08-30 12:52:30,245::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f01980df-e962-4791-814a-b3a6045de8c7`::Granted request
>Thread-1684::DEBUG::2012-08-30 12:52:30,245::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1683::DEBUG::2012-08-30 12:52:30,246::task::817::TaskManager.Task::(resourceAcquired) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1684::DEBUG::2012-08-30 12:52:30,246::task::978::TaskManager.Task::(_decref) Task=`6804587e-fbc2-4bbf-b9ae-3ba82fdab845`::ref 0 aborting False
>Thread-1683::DEBUG::2012-08-30 12:52:30,246::task::978::TaskManager.Task::(_decref) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::ref 1 aborting False
>Thread-1683::INFO::2012-08-30 12:52:30,251::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1683::DEBUG::2012-08-30 12:52:30,251::task::1172::TaskManager.Task::(prepare) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1683::DEBUG::2012-08-30 12:52:30,251::task::588::TaskManager.Task::(_updateState) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::moving from state preparing -> state finished
>Thread-1683::DEBUG::2012-08-30 12:52:30,252::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1683::DEBUG::2012-08-30 12:52:30,252::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1683::DEBUG::2012-08-30 12:52:30,252::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1683::DEBUG::2012-08-30 12:52:30,252::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1683::DEBUG::2012-08-30 12:52:30,253::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1683::DEBUG::2012-08-30 12:52:30,253::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1683::DEBUG::2012-08-30 12:52:30,253::task::978::TaskManager.Task::(_decref) Task=`ff67cb72-f311-4939-9d66-c44eb4ae884a`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:52:32,150::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:52:33,105::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1685::DEBUG::2012-08-30 12:52:33,451::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1685::DEBUG::2012-08-30 12:52:33,451::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:52:35,153::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:38,106::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:38,106::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:40,154::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:43,107::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1687::DEBUG::2012-08-30 12:52:43,494::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1687::DEBUG::2012-08-30 12:52:43,495::task::588::TaskManager.Task::(_updateState) Task=`b4d19073-8cda-4160-9240-6d05063162c8`::moving from state init -> state preparing
>Thread-1687::INFO::2012-08-30 12:52:43,496::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1687::INFO::2012-08-30 12:52:43,496::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1687::DEBUG::2012-08-30 12:52:43,496::task::1172::TaskManager.Task::(prepare) Task=`b4d19073-8cda-4160-9240-6d05063162c8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1687::DEBUG::2012-08-30 12:52:43,496::task::588::TaskManager.Task::(_updateState) Task=`b4d19073-8cda-4160-9240-6d05063162c8`::moving from state preparing -> state finished
>Thread-1687::DEBUG::2012-08-30 12:52:43,497::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1687::DEBUG::2012-08-30 12:52:43,497::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1687::DEBUG::2012-08-30 12:52:43,497::task::978::TaskManager.Task::(_decref) Task=`b4d19073-8cda-4160-9240-6d05063162c8`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:52:45,155::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:45,155::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1689::DEBUG::2012-08-30 12:52:46,681::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1689::DEBUG::2012-08-30 12:52:46,681::task::588::TaskManager.Task::(_updateState) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::moving from state init -> state preparing
>Thread-1689::INFO::2012-08-30 12:52:46,681::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1689::DEBUG::2012-08-30 12:52:46,682::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9d7d506b-600d-4283-8e41-75e8322565e2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1689::DEBUG::2012-08-30 12:52:46,682::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1689::DEBUG::2012-08-30 12:52:46,682::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1689::DEBUG::2012-08-30 12:52:46,682::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9d7d506b-600d-4283-8e41-75e8322565e2`::Granted request
>Thread-1689::DEBUG::2012-08-30 12:52:46,683::task::817::TaskManager.Task::(resourceAcquired) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1689::DEBUG::2012-08-30 12:52:46,683::task::978::TaskManager.Task::(_decref) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::ref 1 aborting False
>Thread-1689::INFO::2012-08-30 12:52:46,686::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1689::DEBUG::2012-08-30 12:52:46,686::task::1172::TaskManager.Task::(prepare) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1689::DEBUG::2012-08-30 12:52:46,686::task::588::TaskManager.Task::(_updateState) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::moving from state preparing -> state finished
>Thread-1689::DEBUG::2012-08-30 12:52:46,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1689::DEBUG::2012-08-30 12:52:46,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1689::DEBUG::2012-08-30 12:52:46,687::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1689::DEBUG::2012-08-30 12:52:46,687::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1689::DEBUG::2012-08-30 12:52:46,688::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1689::DEBUG::2012-08-30 12:52:46,688::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1689::DEBUG::2012-08-30 12:52:46,688::task::978::TaskManager.Task::(_decref) Task=`bca02b51-2fe3-4521-b95f-dd3b1be83425`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:52:48,111::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:50,156::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 12:52:53,113::task::588::TaskManager.Task::(_updateState) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:52:53,113::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:52:53,114::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`76581dbc-e7d3-4998-8fa8-7d4fbb49ec2c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:52:53,114::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:52:53,114::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:52:53,114::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`76581dbc-e7d3-4998-8fa8-7d4fbb49ec2c`::Granted request
>Thread-40::DEBUG::2012-08-30 12:52:53,115::task::817::TaskManager.Task::(resourceAcquired) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:52:53,115::task::978::TaskManager.Task::(_decref) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:52:53,117::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:52:53,119::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:52:53,120::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:52:53,120::task::1172::TaskManager.Task::(prepare) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:52:53,120::task::588::TaskManager.Task::(_updateState) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:52:53,121::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:52:53,121::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:52:53,121::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:52:53,121::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:52:53,122::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:52:53,122::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:52:53,122::task::978::TaskManager.Task::(_decref) Task=`8071c410-0c41-4c1b-9d02-81dcf19b7f47`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:52:53,122::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:53,123::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:53,123::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:52:53,124::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:52:55,157::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:52:55,166::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 12:52:58,125::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1692::DEBUG::2012-08-30 12:52:58,314::task::588::TaskManager.Task::(_updateState) Task=`44d12fb8-aab8-49da-87af-a857e29015e6`::moving from state init -> state preparing
>Thread-1692::INFO::2012-08-30 12:52:58,314::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1692::INFO::2012-08-30 12:52:58,314::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177884101868', 'lastCheck': 1346302369.668272, 'code': 0, 'valid': True}}
>Thread-1692::DEBUG::2012-08-30 12:52:58,315::task::1172::TaskManager.Task::(prepare) Task=`44d12fb8-aab8-49da-87af-a857e29015e6`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177884101868', 'lastCheck': 1346302369.668272, 'code': 0, 'valid': True}}
>Thread-1692::DEBUG::2012-08-30 12:52:58,315::task::588::TaskManager.Task::(_updateState) Task=`44d12fb8-aab8-49da-87af-a857e29015e6`::moving from state preparing -> state finished
>Thread-1692::DEBUG::2012-08-30 12:52:58,315::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1692::DEBUG::2012-08-30 12:52:58,315::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1692::DEBUG::2012-08-30 12:52:58,316::task::978::TaskManager.Task::(_decref) Task=`44d12fb8-aab8-49da-87af-a857e29015e6`::ref 0 aborting False
>Thread-1693::DEBUG::2012-08-30 12:52:59,897::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1693::DEBUG::2012-08-30 12:52:59,898::task::588::TaskManager.Task::(_updateState) Task=`fa25be25-00ce-49c9-aa14-76bf02f4f618`::moving from state init -> state preparing
>Thread-1693::INFO::2012-08-30 12:52:59,898::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1693::INFO::2012-08-30 12:52:59,898::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1693::DEBUG::2012-08-30 12:52:59,899::task::1172::TaskManager.Task::(prepare) Task=`fa25be25-00ce-49c9-aa14-76bf02f4f618`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1693::DEBUG::2012-08-30 12:52:59,899::task::588::TaskManager.Task::(_updateState) Task=`fa25be25-00ce-49c9-aa14-76bf02f4f618`::moving from state preparing -> state finished
>Thread-1693::DEBUG::2012-08-30 12:52:59,899::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1693::DEBUG::2012-08-30 12:52:59,899::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1693::DEBUG::2012-08-30 12:52:59,900::task::978::TaskManager.Task::(_decref) Task=`fa25be25-00ce-49c9-aa14-76bf02f4f618`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:53:00,158::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:00,159::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1694::DEBUG::2012-08-30 12:53:01,535::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1694::DEBUG::2012-08-30 12:53:01,535::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>VM Channels Listener::DEBUG::2012-08-30 12:53:02,171::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1695::DEBUG::2012-08-30 12:53:03,126::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-40::ERROR::2012-08-30 12:53:03,126::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1695::DEBUG::2012-08-30 12:53:03,126::task::588::TaskManager.Task::(_updateState) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::moving from state init -> state preparing
>Thread-1695::INFO::2012-08-30 12:53:03,127::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1695::DEBUG::2012-08-30 12:53:03,127::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`fa469b9a-d82c-4631-9a86-934fb4409847`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1695::DEBUG::2012-08-30 12:53:03,127::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1695::DEBUG::2012-08-30 12:53:03,128::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1695::DEBUG::2012-08-30 12:53:03,128::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`fa469b9a-d82c-4631-9a86-934fb4409847`::Granted request
>Thread-1695::DEBUG::2012-08-30 12:53:03,128::task::817::TaskManager.Task::(resourceAcquired) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1695::DEBUG::2012-08-30 12:53:03,128::task::978::TaskManager.Task::(_decref) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::ref 1 aborting False
>Thread-1695::INFO::2012-08-30 12:53:03,131::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1695::DEBUG::2012-08-30 12:53:03,132::task::1172::TaskManager.Task::(prepare) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1695::DEBUG::2012-08-30 12:53:03,132::task::588::TaskManager.Task::(_updateState) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::moving from state preparing -> state finished
>Thread-1695::DEBUG::2012-08-30 12:53:03,132::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1695::DEBUG::2012-08-30 12:53:03,132::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1695::DEBUG::2012-08-30 12:53:03,133::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1695::DEBUG::2012-08-30 12:53:03,133::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1695::DEBUG::2012-08-30 12:53:03,133::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1695::DEBUG::2012-08-30 12:53:03,133::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1695::DEBUG::2012-08-30 12:53:03,134::task::978::TaskManager.Task::(_decref) Task=`a08a8fac-39fd-4e8c-b1b1-592dd7fae846`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:53:05,159::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:08,127::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:08,128::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:10,160::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:13,129::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:15,161::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:15,162::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1698::DEBUG::2012-08-30 12:53:16,340::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1698::DEBUG::2012-08-30 12:53:16,340::task::588::TaskManager.Task::(_updateState) Task=`387fe826-f739-42af-9ad9-5b6b94790202`::moving from state init -> state preparing
>Thread-1698::INFO::2012-08-30 12:53:16,340::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1698::INFO::2012-08-30 12:53:16,341::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1698::DEBUG::2012-08-30 12:53:16,341::task::1172::TaskManager.Task::(prepare) Task=`387fe826-f739-42af-9ad9-5b6b94790202`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1698::DEBUG::2012-08-30 12:53:16,341::task::588::TaskManager.Task::(_updateState) Task=`387fe826-f739-42af-9ad9-5b6b94790202`::moving from state preparing -> state finished
>Thread-1698::DEBUG::2012-08-30 12:53:16,341::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1698::DEBUG::2012-08-30 12:53:16,342::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1698::DEBUG::2012-08-30 12:53:16,342::task::978::TaskManager.Task::(_decref) Task=`387fe826-f739-42af-9ad9-5b6b94790202`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:53:18,130::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1700::DEBUG::2012-08-30 12:53:19,515::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1700::DEBUG::2012-08-30 12:53:19,515::task::588::TaskManager.Task::(_updateState) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::moving from state init -> state preparing
>Thread-1700::INFO::2012-08-30 12:53:19,515::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1700::DEBUG::2012-08-30 12:53:19,516::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d6f1f642-d46e-4df5-b13e-19fe5e6fe111`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1700::DEBUG::2012-08-30 12:53:19,516::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1700::DEBUG::2012-08-30 12:53:19,516::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1700::DEBUG::2012-08-30 12:53:19,517::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d6f1f642-d46e-4df5-b13e-19fe5e6fe111`::Granted request
>Thread-1700::DEBUG::2012-08-30 12:53:19,517::task::817::TaskManager.Task::(resourceAcquired) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1700::DEBUG::2012-08-30 12:53:19,517::task::978::TaskManager.Task::(_decref) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::ref 1 aborting False
>Thread-1700::INFO::2012-08-30 12:53:19,520::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1700::DEBUG::2012-08-30 12:53:19,520::task::1172::TaskManager.Task::(prepare) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1700::DEBUG::2012-08-30 12:53:19,521::task::588::TaskManager.Task::(_updateState) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::moving from state preparing -> state finished
>Thread-1700::DEBUG::2012-08-30 12:53:19,521::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1700::DEBUG::2012-08-30 12:53:19,521::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1700::DEBUG::2012-08-30 12:53:19,521::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1700::DEBUG::2012-08-30 12:53:19,522::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1700::DEBUG::2012-08-30 12:53:19,522::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1700::DEBUG::2012-08-30 12:53:19,522::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1700::DEBUG::2012-08-30 12:53:19,522::task::978::TaskManager.Task::(_decref) Task=`f43df55a-c48c-457c-8537-93c2369ea717`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:53:20,163::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:23,132::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:23,132::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:25,164::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:53:25,191::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1702::DEBUG::2012-08-30 12:53:27,553::task::588::TaskManager.Task::(_updateState) Task=`c62166f2-de18-47b0-b9b6-3d26fe63ab29`::moving from state init -> state preparing
>Thread-1702::INFO::2012-08-30 12:53:27,553::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1702::INFO::2012-08-30 12:53:27,553::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016348361969', 'lastCheck': 1346302399.6880889, 'code': 0, 'valid': True}}
>Thread-1702::DEBUG::2012-08-30 12:53:27,554::task::1172::TaskManager.Task::(prepare) Task=`c62166f2-de18-47b0-b9b6-3d26fe63ab29`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016348361969', 'lastCheck': 1346302399.6880889, 'code': 0, 'valid': True}}
>Thread-1702::DEBUG::2012-08-30 12:53:27,554::task::588::TaskManager.Task::(_updateState) Task=`c62166f2-de18-47b0-b9b6-3d26fe63ab29`::moving from state preparing -> state finished
>Thread-1702::DEBUG::2012-08-30 12:53:27,554::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1702::DEBUG::2012-08-30 12:53:27,554::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1702::DEBUG::2012-08-30 12:53:27,554::task::978::TaskManager.Task::(_decref) Task=`c62166f2-de18-47b0-b9b6-3d26fe63ab29`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:53:28,133::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:53:30,165::task::588::TaskManager.Task::(_updateState) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:53:30,165::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:53:30,165::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bbbff8cd-08ce-4e81-a77c-1a3f38a80dbe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:53:30,166::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:53:30,166::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:53:30,166::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bbbff8cd-08ce-4e81-a77c-1a3f38a80dbe`::Granted request
>Thread-104::DEBUG::2012-08-30 12:53:30,166::task::817::TaskManager.Task::(resourceAcquired) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:53:30,167::task::978::TaskManager.Task::(_decref) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:53:30,168::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:53:30,170::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:53:30,171::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:53:30,172::task::1172::TaskManager.Task::(prepare) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:53:30,172::task::588::TaskManager.Task::(_updateState) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:53:30,172::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:53:30,172::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:53:30,173::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:53:30,173::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:53:30,173::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:53:30,173::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:53:30,173::task::978::TaskManager.Task::(_decref) Task=`8fe9c4cd-46fc-4ca9-b167-e7c552abde0d`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:53:30,174::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:30,174::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:30,175::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:30,175::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1703::DEBUG::2012-08-30 12:53:30,743::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1703::DEBUG::2012-08-30 12:53:30,744::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>VM Channels Listener::DEBUG::2012-08-30 12:53:32,198::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1704::DEBUG::2012-08-30 12:53:32,329::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1704::DEBUG::2012-08-30 12:53:32,329::task::588::TaskManager.Task::(_updateState) Task=`e9e9e3f5-95a0-42bd-b277-c8d624eb415c`::moving from state init -> state preparing
>Thread-1704::INFO::2012-08-30 12:53:32,329::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1704::INFO::2012-08-30 12:53:32,329::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1704::DEBUG::2012-08-30 12:53:32,330::task::1172::TaskManager.Task::(prepare) Task=`e9e9e3f5-95a0-42bd-b277-c8d624eb415c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1704::DEBUG::2012-08-30 12:53:32,330::task::588::TaskManager.Task::(_updateState) Task=`e9e9e3f5-95a0-42bd-b277-c8d624eb415c`::moving from state preparing -> state finished
>Thread-1704::DEBUG::2012-08-30 12:53:32,330::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1704::DEBUG::2012-08-30 12:53:32,330::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1704::DEBUG::2012-08-30 12:53:32,331::task::978::TaskManager.Task::(_decref) Task=`e9e9e3f5-95a0-42bd-b277-c8d624eb415c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:53:33,134::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:35,176::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1705::DEBUG::2012-08-30 12:53:35,506::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1705::DEBUG::2012-08-30 12:53:35,507::task::588::TaskManager.Task::(_updateState) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::moving from state init -> state preparing
>Thread-1705::INFO::2012-08-30 12:53:35,508::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1705::DEBUG::2012-08-30 12:53:35,508::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c5fe1763-638c-4320-ab1b-2a1b4d9a3237`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1705::DEBUG::2012-08-30 12:53:35,508::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1705::DEBUG::2012-08-30 12:53:35,508::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1705::DEBUG::2012-08-30 12:53:35,509::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c5fe1763-638c-4320-ab1b-2a1b4d9a3237`::Granted request
>Thread-1705::DEBUG::2012-08-30 12:53:35,509::task::817::TaskManager.Task::(resourceAcquired) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1705::DEBUG::2012-08-30 12:53:35,509::task::978::TaskManager.Task::(_decref) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::ref 1 aborting False
>Thread-1705::INFO::2012-08-30 12:53:35,512::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1705::DEBUG::2012-08-30 12:53:35,512::task::1172::TaskManager.Task::(prepare) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1705::DEBUG::2012-08-30 12:53:35,513::task::588::TaskManager.Task::(_updateState) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::moving from state preparing -> state finished
>Thread-1705::DEBUG::2012-08-30 12:53:35,513::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1705::DEBUG::2012-08-30 12:53:35,513::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1705::DEBUG::2012-08-30 12:53:35,513::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1705::DEBUG::2012-08-30 12:53:35,514::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1705::DEBUG::2012-08-30 12:53:35,514::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1705::DEBUG::2012-08-30 12:53:35,514::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1705::DEBUG::2012-08-30 12:53:35,514::task::978::TaskManager.Task::(_decref) Task=`d66628df-f7fe-4c46-a9cc-0bcf221e2f60`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:53:38,136::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:38,136::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:40,177::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:43,137::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:45,178::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:45,178::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:48,139::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1709::DEBUG::2012-08-30 12:53:48,855::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1709::DEBUG::2012-08-30 12:53:48,856::task::588::TaskManager.Task::(_updateState) Task=`97dbf53b-308e-4cd3-b6bf-fbd3fd9e04e8`::moving from state init -> state preparing
>Thread-1709::INFO::2012-08-30 12:53:48,856::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1709::INFO::2012-08-30 12:53:48,857::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1709::DEBUG::2012-08-30 12:53:48,857::task::1172::TaskManager.Task::(prepare) Task=`97dbf53b-308e-4cd3-b6bf-fbd3fd9e04e8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1709::DEBUG::2012-08-30 12:53:48,857::task::588::TaskManager.Task::(_updateState) Task=`97dbf53b-308e-4cd3-b6bf-fbd3fd9e04e8`::moving from state preparing -> state finished
>Thread-1709::DEBUG::2012-08-30 12:53:48,857::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1709::DEBUG::2012-08-30 12:53:48,857::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1709::DEBUG::2012-08-30 12:53:48,858::task::978::TaskManager.Task::(_decref) Task=`97dbf53b-308e-4cd3-b6bf-fbd3fd9e04e8`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:53:50,179::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1711::DEBUG::2012-08-30 12:53:52,058::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1711::DEBUG::2012-08-30 12:53:52,059::task::588::TaskManager.Task::(_updateState) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::moving from state init -> state preparing
>Thread-1711::INFO::2012-08-30 12:53:52,059::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1711::DEBUG::2012-08-30 12:53:52,059::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`48b44b7d-9a35-41cf-b21f-96f160f6a8de`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1711::DEBUG::2012-08-30 12:53:52,059::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1711::DEBUG::2012-08-30 12:53:52,060::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1711::DEBUG::2012-08-30 12:53:52,060::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`48b44b7d-9a35-41cf-b21f-96f160f6a8de`::Granted request
>Thread-1711::DEBUG::2012-08-30 12:53:52,060::task::817::TaskManager.Task::(resourceAcquired) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1711::DEBUG::2012-08-30 12:53:52,061::task::978::TaskManager.Task::(_decref) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::ref 1 aborting False
>Thread-1711::INFO::2012-08-30 12:53:52,063::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1711::DEBUG::2012-08-30 12:53:52,064::task::1172::TaskManager.Task::(prepare) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1711::DEBUG::2012-08-30 12:53:52,064::task::588::TaskManager.Task::(_updateState) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::moving from state preparing -> state finished
>Thread-1711::DEBUG::2012-08-30 12:53:52,064::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1711::DEBUG::2012-08-30 12:53:52,064::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1711::DEBUG::2012-08-30 12:53:52,065::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1711::DEBUG::2012-08-30 12:53:52,065::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1711::DEBUG::2012-08-30 12:53:52,065::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1711::DEBUG::2012-08-30 12:53:52,065::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1711::DEBUG::2012-08-30 12:53:52,066::task::978::TaskManager.Task::(_decref) Task=`0481d8fb-4ba9-451e-890e-590cd0d3f04a`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 12:53:53,140::task::588::TaskManager.Task::(_updateState) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:53:53,140::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:53:53,141::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`df8a4022-932a-43be-ad6d-35cb7779e7be`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:53:53,141::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:53:53,141::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:53:53,141::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`df8a4022-932a-43be-ad6d-35cb7779e7be`::Granted request
>Thread-40::DEBUG::2012-08-30 12:53:53,142::task::817::TaskManager.Task::(resourceAcquired) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:53:53,142::task::978::TaskManager.Task::(_decref) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:53:53,144::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:53:53,146::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:53:53,147::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:53:53,147::task::1172::TaskManager.Task::(prepare) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:53:53,147::task::588::TaskManager.Task::(_updateState) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:53:53,148::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:53:53,148::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:53:53,148::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:53:53,148::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:53:53,149::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:53:53,149::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:53:53,149::task::978::TaskManager.Task::(_decref) Task=`555603ce-3a6b-4af3-b44c-441b84407fba`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:53:53,149::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:53,150::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:53,150::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:53:53,151::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:53:55,180::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:53:55,216::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1712::DEBUG::2012-08-30 12:53:56,487::task::588::TaskManager.Task::(_updateState) Task=`5394ee6e-c1b9-4063-9a53-b201be6aec77`::moving from state init -> state preparing
>Thread-1712::INFO::2012-08-30 12:53:56,487::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1712::INFO::2012-08-30 12:53:56,487::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177502632141', 'lastCheck': 1346302429.7076781, 'code': 0, 'valid': True}}
>Thread-1712::DEBUG::2012-08-30 12:53:56,488::task::1172::TaskManager.Task::(prepare) Task=`5394ee6e-c1b9-4063-9a53-b201be6aec77`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177502632141', 'lastCheck': 1346302429.7076781, 'code': 0, 'valid': True}}
>Thread-1712::DEBUG::2012-08-30 12:53:56,488::task::588::TaskManager.Task::(_updateState) Task=`5394ee6e-c1b9-4063-9a53-b201be6aec77`::moving from state preparing -> state finished
>Thread-1712::DEBUG::2012-08-30 12:53:56,488::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1712::DEBUG::2012-08-30 12:53:56,488::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1712::DEBUG::2012-08-30 12:53:56,488::task::978::TaskManager.Task::(_decref) Task=`5394ee6e-c1b9-4063-9a53-b201be6aec77`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:53:58,152::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1713::DEBUG::2012-08-30 12:53:59,693::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1713::DEBUG::2012-08-30 12:53:59,694::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:54:00,181::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:00,182::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:54:02,222::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:54:03,153::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1715::DEBUG::2012-08-30 12:54:04,965::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1715::DEBUG::2012-08-30 12:54:04,966::task::588::TaskManager.Task::(_updateState) Task=`961d92c0-d141-41a2-9e66-3c85efe06717`::moving from state init -> state preparing
>Thread-1715::INFO::2012-08-30 12:54:04,966::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1715::INFO::2012-08-30 12:54:04,967::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1715::DEBUG::2012-08-30 12:54:04,967::task::1172::TaskManager.Task::(prepare) Task=`961d92c0-d141-41a2-9e66-3c85efe06717`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1715::DEBUG::2012-08-30 12:54:04,967::task::588::TaskManager.Task::(_updateState) Task=`961d92c0-d141-41a2-9e66-3c85efe06717`::moving from state preparing -> state finished
>Thread-1715::DEBUG::2012-08-30 12:54:04,967::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1715::DEBUG::2012-08-30 12:54:04,967::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1715::DEBUG::2012-08-30 12:54:04,968::task::978::TaskManager.Task::(_decref) Task=`961d92c0-d141-41a2-9e66-3c85efe06717`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:54:05,183::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1716::DEBUG::2012-08-30 12:54:08,151::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1716::DEBUG::2012-08-30 12:54:08,151::task::588::TaskManager.Task::(_updateState) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::moving from state init -> state preparing
>Thread-1716::INFO::2012-08-30 12:54:08,152::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1716::DEBUG::2012-08-30 12:54:08,152::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`eb532913-abf1-4ffe-9ed7-009dabdb47ae`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1716::DEBUG::2012-08-30 12:54:08,152::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1716::DEBUG::2012-08-30 12:54:08,153::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1716::DEBUG::2012-08-30 12:54:08,153::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`eb532913-abf1-4ffe-9ed7-009dabdb47ae`::Granted request
>Thread-1716::DEBUG::2012-08-30 12:54:08,153::task::817::TaskManager.Task::(resourceAcquired) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1716::DEBUG::2012-08-30 12:54:08,154::task::978::TaskManager.Task::(_decref) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::ref 1 aborting False
>Thread-40::ERROR::2012-08-30 12:54:08,154::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:08,156::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1716::INFO::2012-08-30 12:54:08,158::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1716::DEBUG::2012-08-30 12:54:08,158::task::1172::TaskManager.Task::(prepare) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1716::DEBUG::2012-08-30 12:54:08,158::task::588::TaskManager.Task::(_updateState) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::moving from state preparing -> state finished
>Thread-1716::DEBUG::2012-08-30 12:54:08,158::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1716::DEBUG::2012-08-30 12:54:08,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1716::DEBUG::2012-08-30 12:54:08,159::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1716::DEBUG::2012-08-30 12:54:08,159::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1716::DEBUG::2012-08-30 12:54:08,159::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1716::DEBUG::2012-08-30 12:54:08,160::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1716::DEBUG::2012-08-30 12:54:08,160::task::978::TaskManager.Task::(_decref) Task=`4a5628de-5beb-419a-833b-b84250ee5bd9`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:54:10,183::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:13,158::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:15,184::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:15,185::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:18,160::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:20,186::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1720::DEBUG::2012-08-30 12:54:21,485::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1720::DEBUG::2012-08-30 12:54:21,485::task::588::TaskManager.Task::(_updateState) Task=`60ec2e70-0b9b-4404-b09d-b16636716b9a`::moving from state init -> state preparing
>Thread-1720::INFO::2012-08-30 12:54:21,486::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1720::INFO::2012-08-30 12:54:21,486::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1720::DEBUG::2012-08-30 12:54:21,486::task::1172::TaskManager.Task::(prepare) Task=`60ec2e70-0b9b-4404-b09d-b16636716b9a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1720::DEBUG::2012-08-30 12:54:21,486::task::588::TaskManager.Task::(_updateState) Task=`60ec2e70-0b9b-4404-b09d-b16636716b9a`::moving from state preparing -> state finished
>Thread-1720::DEBUG::2012-08-30 12:54:21,487::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1720::DEBUG::2012-08-30 12:54:21,487::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1720::DEBUG::2012-08-30 12:54:21,487::task::978::TaskManager.Task::(_decref) Task=`60ec2e70-0b9b-4404-b09d-b16636716b9a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:54:23,161::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:23,162::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1721::DEBUG::2012-08-30 12:54:24,667::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1722::DEBUG::2012-08-30 12:54:24,667::task::588::TaskManager.Task::(_updateState) Task=`aed629d9-9df2-495c-83e7-664aeb51a2a2`::moving from state init -> state preparing
>Thread-1721::DEBUG::2012-08-30 12:54:24,668::task::588::TaskManager.Task::(_updateState) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::moving from state init -> state preparing
>Thread-1722::INFO::2012-08-30 12:54:24,668::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1721::INFO::2012-08-30 12:54:24,668::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1722::INFO::2012-08-30 12:54:24,668::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346302459.7277031, 'code': 0, 'valid': True}}
>Thread-1721::DEBUG::2012-08-30 12:54:24,669::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`11742f64-567e-46cb-8295-6e1e97a0fe9b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1722::DEBUG::2012-08-30 12:54:24,669::task::1172::TaskManager.Task::(prepare) Task=`aed629d9-9df2-495c-83e7-664aeb51a2a2`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346302459.7277031, 'code': 0, 'valid': True}}
>Thread-1721::DEBUG::2012-08-30 12:54:24,669::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1722::DEBUG::2012-08-30 12:54:24,669::task::588::TaskManager.Task::(_updateState) Task=`aed629d9-9df2-495c-83e7-664aeb51a2a2`::moving from state preparing -> state finished
>Thread-1721::DEBUG::2012-08-30 12:54:24,669::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1722::DEBUG::2012-08-30 12:54:24,670::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1721::DEBUG::2012-08-30 12:54:24,670::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`11742f64-567e-46cb-8295-6e1e97a0fe9b`::Granted request
>Thread-1722::DEBUG::2012-08-30 12:54:24,670::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1721::DEBUG::2012-08-30 12:54:24,670::task::817::TaskManager.Task::(resourceAcquired) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1722::DEBUG::2012-08-30 12:54:24,671::task::978::TaskManager.Task::(_decref) Task=`aed629d9-9df2-495c-83e7-664aeb51a2a2`::ref 0 aborting False
>Thread-1721::DEBUG::2012-08-30 12:54:24,671::task::978::TaskManager.Task::(_decref) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::ref 1 aborting False
>Thread-1721::INFO::2012-08-30 12:54:24,676::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1721::DEBUG::2012-08-30 12:54:24,676::task::1172::TaskManager.Task::(prepare) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1721::DEBUG::2012-08-30 12:54:24,676::task::588::TaskManager.Task::(_updateState) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::moving from state preparing -> state finished
>Thread-1721::DEBUG::2012-08-30 12:54:24,676::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1721::DEBUG::2012-08-30 12:54:24,676::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1721::DEBUG::2012-08-30 12:54:24,677::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1721::DEBUG::2012-08-30 12:54:24,677::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1721::DEBUG::2012-08-30 12:54:24,677::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1721::DEBUG::2012-08-30 12:54:24,677::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1721::DEBUG::2012-08-30 12:54:24,678::task::978::TaskManager.Task::(_decref) Task=`0bb93a72-e759-4934-a5aa-a0c33c393205`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:54:25,187::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:54:25,239::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1723::DEBUG::2012-08-30 12:54:27,863::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1723::DEBUG::2012-08-30 12:54:27,863::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 12:54:28,163::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:54:30,188::task::588::TaskManager.Task::(_updateState) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:54:30,188::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:54:30,188::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`fb80bf79-47d0-497b-a2c7-6926f802d73f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:54:30,189::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:54:30,189::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:54:30,189::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`fb80bf79-47d0-497b-a2c7-6926f802d73f`::Granted request
>Thread-104::DEBUG::2012-08-30 12:54:30,190::task::817::TaskManager.Task::(resourceAcquired) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:54:30,190::task::978::TaskManager.Task::(_decref) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:54:30,191::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:54:30,193::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:54:30,195::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:54:30,195::task::1172::TaskManager.Task::(prepare) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:54:30,195::task::588::TaskManager.Task::(_updateState) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:54:30,195::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:54:30,195::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:54:30,196::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:54:30,196::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:54:30,196::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:54:30,196::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:54:30,197::task::978::TaskManager.Task::(_decref) Task=`f2f79ec7-7669-4d41-80c9-792f0a3a9f7f`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:54:30,197::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:30,198::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:30,198::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:30,198::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:54:32,246::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:54:33,164::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:35,199::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1725::DEBUG::2012-08-30 12:54:37,891::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1725::DEBUG::2012-08-30 12:54:37,892::task::588::TaskManager.Task::(_updateState) Task=`46e09f0e-9d6e-4cc1-ab8f-fe41f20f88d1`::moving from state init -> state preparing
>Thread-1725::INFO::2012-08-30 12:54:37,892::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1725::INFO::2012-08-30 12:54:37,892::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1725::DEBUG::2012-08-30 12:54:37,892::task::1172::TaskManager.Task::(prepare) Task=`46e09f0e-9d6e-4cc1-ab8f-fe41f20f88d1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1725::DEBUG::2012-08-30 12:54:37,893::task::588::TaskManager.Task::(_updateState) Task=`46e09f0e-9d6e-4cc1-ab8f-fe41f20f88d1`::moving from state preparing -> state finished
>Thread-1725::DEBUG::2012-08-30 12:54:37,893::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1725::DEBUG::2012-08-30 12:54:37,893::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1725::DEBUG::2012-08-30 12:54:37,893::task::978::TaskManager.Task::(_decref) Task=`46e09f0e-9d6e-4cc1-ab8f-fe41f20f88d1`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:54:38,165::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:38,166::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:40,200::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1727::DEBUG::2012-08-30 12:54:41,086::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1727::DEBUG::2012-08-30 12:54:41,086::task::588::TaskManager.Task::(_updateState) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::moving from state init -> state preparing
>Thread-1727::INFO::2012-08-30 12:54:41,087::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1727::DEBUG::2012-08-30 12:54:41,087::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ec3d6554-3d37-49f4-8f2e-de4828b8a3bf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1727::DEBUG::2012-08-30 12:54:41,087::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1727::DEBUG::2012-08-30 12:54:41,088::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1727::DEBUG::2012-08-30 12:54:41,088::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ec3d6554-3d37-49f4-8f2e-de4828b8a3bf`::Granted request
>Thread-1727::DEBUG::2012-08-30 12:54:41,088::task::817::TaskManager.Task::(resourceAcquired) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1727::DEBUG::2012-08-30 12:54:41,088::task::978::TaskManager.Task::(_decref) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::ref 1 aborting False
>Thread-1727::INFO::2012-08-30 12:54:41,091::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1727::DEBUG::2012-08-30 12:54:41,092::task::1172::TaskManager.Task::(prepare) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1727::DEBUG::2012-08-30 12:54:41,092::task::588::TaskManager.Task::(_updateState) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::moving from state preparing -> state finished
>Thread-1727::DEBUG::2012-08-30 12:54:41,092::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1727::DEBUG::2012-08-30 12:54:41,092::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1727::DEBUG::2012-08-30 12:54:41,093::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1727::DEBUG::2012-08-30 12:54:41,093::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1727::DEBUG::2012-08-30 12:54:41,093::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1727::DEBUG::2012-08-30 12:54:41,093::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1727::DEBUG::2012-08-30 12:54:41,093::task::978::TaskManager.Task::(_decref) Task=`9fbb2e88-18ed-4a4f-a036-cb898676be1e`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:54:43,167::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:45,201::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:45,202::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:48,168::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:54:50,203::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 12:54:53,169::task::588::TaskManager.Task::(_updateState) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:54:53,169::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:54:53,170::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4608a04f-68cc-4a87-b7ee-f46adc27d724`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:54:53,170::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:54:53,171::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:54:53,171::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4608a04f-68cc-4a87-b7ee-f46adc27d724`::Granted request
>Thread-40::DEBUG::2012-08-30 12:54:53,171::task::817::TaskManager.Task::(resourceAcquired) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:54:53,171::task::978::TaskManager.Task::(_decref) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:54:53,173::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:54:53,175::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:54:53,176::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:54:53,177::task::1172::TaskManager.Task::(prepare) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:54:53,177::task::588::TaskManager.Task::(_updateState) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:54:53,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:54:53,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:54:53,178::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:54:53,178::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:54:53,178::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:54:53,178::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:54:53,179::task::978::TaskManager.Task::(_decref) Task=`b5e88903-7919-476a-b257-a3d75ddd5bbd`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:54:53,179::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:53,179::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:53,180::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:54:53,180::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1730::DEBUG::2012-08-30 12:54:54,529::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1730::DEBUG::2012-08-30 12:54:54,529::task::588::TaskManager.Task::(_updateState) Task=`938a4193-075c-4dcf-bef3-f168c8d278eb`::moving from state init -> state preparing
>Thread-1730::INFO::2012-08-30 12:54:54,530::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1730::INFO::2012-08-30 12:54:54,530::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1730::DEBUG::2012-08-30 12:54:54,530::task::1172::TaskManager.Task::(prepare) Task=`938a4193-075c-4dcf-bef3-f168c8d278eb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1730::DEBUG::2012-08-30 12:54:54,530::task::588::TaskManager.Task::(_updateState) Task=`938a4193-075c-4dcf-bef3-f168c8d278eb`::moving from state preparing -> state finished
>Thread-1730::DEBUG::2012-08-30 12:54:54,531::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1730::DEBUG::2012-08-30 12:54:54,531::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1730::DEBUG::2012-08-30 12:54:54,531::task::978::TaskManager.Task::(_decref) Task=`938a4193-075c-4dcf-bef3-f168c8d278eb`::ref 0 aborting False
>Thread-1731::DEBUG::2012-08-30 12:54:54,532::task::588::TaskManager.Task::(_updateState) Task=`1217bca7-b05c-413d-a105-909846ee2064`::moving from state init -> state preparing
>Thread-1731::INFO::2012-08-30 12:54:54,532::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1731::INFO::2012-08-30 12:54:54,533::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00196695327759', 'lastCheck': 1346302489.7485011, 'code': 0, 'valid': True}}
>Thread-1731::DEBUG::2012-08-30 12:54:54,533::task::1172::TaskManager.Task::(prepare) Task=`1217bca7-b05c-413d-a105-909846ee2064`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00196695327759', 'lastCheck': 1346302489.7485011, 'code': 0, 'valid': True}}
>Thread-1731::DEBUG::2012-08-30 12:54:54,533::task::588::TaskManager.Task::(_updateState) Task=`1217bca7-b05c-413d-a105-909846ee2064`::moving from state preparing -> state finished
>Thread-1731::DEBUG::2012-08-30 12:54:54,533::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1731::DEBUG::2012-08-30 12:54:54,534::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1731::DEBUG::2012-08-30 12:54:54,534::task::978::TaskManager.Task::(_decref) Task=`1217bca7-b05c-413d-a105-909846ee2064`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:54:55,204::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:54:55,266::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1732::DEBUG::2012-08-30 12:54:57,875::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1732::DEBUG::2012-08-30 12:54:57,876::task::588::TaskManager.Task::(_updateState) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::moving from state init -> state preparing
>Thread-1732::INFO::2012-08-30 12:54:57,876::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1732::DEBUG::2012-08-30 12:54:57,876::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`99ca25af-061c-4a07-946e-557f65600d22`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1732::DEBUG::2012-08-30 12:54:57,876::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1732::DEBUG::2012-08-30 12:54:57,877::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1732::DEBUG::2012-08-30 12:54:57,877::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`99ca25af-061c-4a07-946e-557f65600d22`::Granted request
>Thread-1732::DEBUG::2012-08-30 12:54:57,877::task::817::TaskManager.Task::(resourceAcquired) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1732::DEBUG::2012-08-30 12:54:57,878::task::978::TaskManager.Task::(_decref) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::ref 1 aborting False
>Thread-1733::DEBUG::2012-08-30 12:54:57,879::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1733::DEBUG::2012-08-30 12:54:57,879::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1732::INFO::2012-08-30 12:54:57,883::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1732::DEBUG::2012-08-30 12:54:57,883::task::1172::TaskManager.Task::(prepare) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1732::DEBUG::2012-08-30 12:54:57,883::task::588::TaskManager.Task::(_updateState) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::moving from state preparing -> state finished
>Thread-1732::DEBUG::2012-08-30 12:54:57,884::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1732::DEBUG::2012-08-30 12:54:57,884::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1732::DEBUG::2012-08-30 12:54:57,884::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1732::DEBUG::2012-08-30 12:54:57,884::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1732::DEBUG::2012-08-30 12:54:57,885::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1732::DEBUG::2012-08-30 12:54:57,885::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1732::DEBUG::2012-08-30 12:54:57,885::task::978::TaskManager.Task::(_decref) Task=`a71f70b0-c047-46a1-a0e1-4b9122584217`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:54:58,181::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:00,205::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:00,205::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:55:02,274::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:55:03,183::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:05,206::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:08,185::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:08,185::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:10,207::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1736::DEBUG::2012-08-30 12:55:11,131::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1736::DEBUG::2012-08-30 12:55:11,132::task::588::TaskManager.Task::(_updateState) Task=`6ecf885b-8154-42fa-b236-f56e4c36ef99`::moving from state init -> state preparing
>Thread-1736::INFO::2012-08-30 12:55:11,132::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1736::INFO::2012-08-30 12:55:11,132::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1736::DEBUG::2012-08-30 12:55:11,132::task::1172::TaskManager.Task::(prepare) Task=`6ecf885b-8154-42fa-b236-f56e4c36ef99`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1736::DEBUG::2012-08-30 12:55:11,133::task::588::TaskManager.Task::(_updateState) Task=`6ecf885b-8154-42fa-b236-f56e4c36ef99`::moving from state preparing -> state finished
>Thread-1736::DEBUG::2012-08-30 12:55:11,133::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1736::DEBUG::2012-08-30 12:55:11,133::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1736::DEBUG::2012-08-30 12:55:11,133::task::978::TaskManager.Task::(_decref) Task=`6ecf885b-8154-42fa-b236-f56e4c36ef99`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:55:13,186::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1738::DEBUG::2012-08-30 12:55:14,327::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1738::DEBUG::2012-08-30 12:55:14,328::task::588::TaskManager.Task::(_updateState) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::moving from state init -> state preparing
>Thread-1738::INFO::2012-08-30 12:55:14,328::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1738::DEBUG::2012-08-30 12:55:14,328::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ba971fc1-a04b-4c46-a6e0-6d8707060c2e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1738::DEBUG::2012-08-30 12:55:14,328::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1738::DEBUG::2012-08-30 12:55:14,329::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1738::DEBUG::2012-08-30 12:55:14,329::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ba971fc1-a04b-4c46-a6e0-6d8707060c2e`::Granted request
>Thread-1738::DEBUG::2012-08-30 12:55:14,329::task::817::TaskManager.Task::(resourceAcquired) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1738::DEBUG::2012-08-30 12:55:14,329::task::978::TaskManager.Task::(_decref) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::ref 1 aborting False
>Thread-1738::INFO::2012-08-30 12:55:14,332::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1738::DEBUG::2012-08-30 12:55:14,333::task::1172::TaskManager.Task::(prepare) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1738::DEBUG::2012-08-30 12:55:14,333::task::588::TaskManager.Task::(_updateState) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::moving from state preparing -> state finished
>Thread-1738::DEBUG::2012-08-30 12:55:14,333::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1738::DEBUG::2012-08-30 12:55:14,333::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1738::DEBUG::2012-08-30 12:55:14,334::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1738::DEBUG::2012-08-30 12:55:14,334::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1738::DEBUG::2012-08-30 12:55:14,334::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1738::DEBUG::2012-08-30 12:55:14,334::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1738::DEBUG::2012-08-30 12:55:14,335::task::978::TaskManager.Task::(_decref) Task=`ce077bc0-d167-4fad-9801-9bc8aa753c6f`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:55:15,208::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:15,208::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:18,187::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:20,209::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:23,189::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:23,189::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1740::DEBUG::2012-08-30 12:55:23,950::task::588::TaskManager.Task::(_updateState) Task=`3531e501-5d39-419c-a175-98a5051661eb`::moving from state init -> state preparing
>Thread-1740::INFO::2012-08-30 12:55:23,951::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1740::INFO::2012-08-30 12:55:23,951::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016188621521', 'lastCheck': 1346302519.7696941, 'code': 0, 'valid': True}}
>Thread-1740::DEBUG::2012-08-30 12:55:23,951::task::1172::TaskManager.Task::(prepare) Task=`3531e501-5d39-419c-a175-98a5051661eb`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016188621521', 'lastCheck': 1346302519.7696941, 'code': 0, 'valid': True}}
>Thread-1740::DEBUG::2012-08-30 12:55:23,951::task::588::TaskManager.Task::(_updateState) Task=`3531e501-5d39-419c-a175-98a5051661eb`::moving from state preparing -> state finished
>Thread-1740::DEBUG::2012-08-30 12:55:23,952::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1740::DEBUG::2012-08-30 12:55:23,952::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1740::DEBUG::2012-08-30 12:55:23,952::task::978::TaskManager.Task::(_decref) Task=`3531e501-5d39-419c-a175-98a5051661eb`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:55:25,210::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:55:25,295::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1741::DEBUG::2012-08-30 12:55:27,258::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1741::DEBUG::2012-08-30 12:55:27,258::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1742::DEBUG::2012-08-30 12:55:27,260::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1742::DEBUG::2012-08-30 12:55:27,260::task::588::TaskManager.Task::(_updateState) Task=`6fe4c3ab-add0-4f53-9486-74cc41bd373d`::moving from state init -> state preparing
>Thread-1742::INFO::2012-08-30 12:55:27,261::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1742::INFO::2012-08-30 12:55:27,261::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1742::DEBUG::2012-08-30 12:55:27,261::task::1172::TaskManager.Task::(prepare) Task=`6fe4c3ab-add0-4f53-9486-74cc41bd373d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1742::DEBUG::2012-08-30 12:55:27,261::task::588::TaskManager.Task::(_updateState) Task=`6fe4c3ab-add0-4f53-9486-74cc41bd373d`::moving from state preparing -> state finished
>Thread-1742::DEBUG::2012-08-30 12:55:27,262::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1742::DEBUG::2012-08-30 12:55:27,262::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1742::DEBUG::2012-08-30 12:55:27,262::task::978::TaskManager.Task::(_decref) Task=`6fe4c3ab-add0-4f53-9486-74cc41bd373d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:55:28,191::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-26::DEBUG::2012-08-30 12:55:29,769::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 12:55:29,770::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-26::DEBUG::2012-08-30 12:55:29,770::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None)
>Thread-26::DEBUG::2012-08-30 12:55:29,891::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = '  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5
>Thread-26::WARNING::2012-08-30 12:55:29,893::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found']
>Thread-26::DEBUG::2012-08-30 12:55:29,893::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-26::DEBUG::2012-08-30 12:55:29,899::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 12:55:29,899::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend
>Thread-26::DEBUG::2012-08-30 12:55:29,902::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d']
>Thread-26::DEBUG::2012-08-30 12:55:29,903::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-26::WARNING::2012-08-30 12:55:29,903::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered
>Thread-26::WARNING::2012-08-30 12:55:29,903::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered
>Thread-104::DEBUG::2012-08-30 12:55:30,211::task::588::TaskManager.Task::(_updateState) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:55:30,212::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:55:30,212::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a53bbf85-da13-4ccd-b431-e7a28ff5c523`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:55:30,212::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:55:30,213::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:55:30,213::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a53bbf85-da13-4ccd-b431-e7a28ff5c523`::Granted request
>Thread-104::DEBUG::2012-08-30 12:55:30,213::task::817::TaskManager.Task::(resourceAcquired) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:55:30,213::task::978::TaskManager.Task::(_decref) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:55:30,215::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:55:30,217::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:55:30,218::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:55:30,219::task::1172::TaskManager.Task::(prepare) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:55:30,219::task::588::TaskManager.Task::(_updateState) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:55:30,219::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:55:30,219::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:55:30,220::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:55:30,220::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:55:30,220::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:55:30,220::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:55:30,220::task::978::TaskManager.Task::(_decref) Task=`cc1548e1-9114-4f03-bdfd-819cfb321c4d`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:55:30,221::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:30,221::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:30,222::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:30,222::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1743::DEBUG::2012-08-30 12:55:30,455::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1743::DEBUG::2012-08-30 12:55:30,456::task::588::TaskManager.Task::(_updateState) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::moving from state init -> state preparing
>Thread-1743::INFO::2012-08-30 12:55:30,456::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1743::DEBUG::2012-08-30 12:55:30,456::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d971836a-854e-422d-8db5-efe4fd9bb2f4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1743::DEBUG::2012-08-30 12:55:30,457::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1743::DEBUG::2012-08-30 12:55:30,457::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1743::DEBUG::2012-08-30 12:55:30,457::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d971836a-854e-422d-8db5-efe4fd9bb2f4`::Granted request
>Thread-1743::DEBUG::2012-08-30 12:55:30,457::task::817::TaskManager.Task::(resourceAcquired) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1743::DEBUG::2012-08-30 12:55:30,458::task::978::TaskManager.Task::(_decref) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::ref 1 aborting False
>Thread-1743::INFO::2012-08-30 12:55:30,461::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1743::DEBUG::2012-08-30 12:55:30,461::task::1172::TaskManager.Task::(prepare) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1743::DEBUG::2012-08-30 12:55:30,461::task::588::TaskManager.Task::(_updateState) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::moving from state preparing -> state finished
>Thread-1743::DEBUG::2012-08-30 12:55:30,461::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1743::DEBUG::2012-08-30 12:55:30,462::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1743::DEBUG::2012-08-30 12:55:30,463::task::978::TaskManager.Task::(_decref) Task=`67de581d-15f9-4e97-b39e-b4a0f81fb333`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:55:32,302::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:55:33,192::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:35,223::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:38,193::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:38,193::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:40,224::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:43,194::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1752::DEBUG::2012-08-30 12:55:43,686::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1752::DEBUG::2012-08-30 12:55:43,686::task::588::TaskManager.Task::(_updateState) Task=`88738fb9-06f6-4fef-9d6f-40b1b82994e2`::moving from state init -> state preparing
>Thread-1752::INFO::2012-08-30 12:55:43,687::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1752::INFO::2012-08-30 12:55:43,687::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1752::DEBUG::2012-08-30 12:55:43,687::task::1172::TaskManager.Task::(prepare) Task=`88738fb9-06f6-4fef-9d6f-40b1b82994e2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1752::DEBUG::2012-08-30 12:55:43,687::task::588::TaskManager.Task::(_updateState) Task=`88738fb9-06f6-4fef-9d6f-40b1b82994e2`::moving from state preparing -> state finished
>Thread-1752::DEBUG::2012-08-30 12:55:43,688::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1752::DEBUG::2012-08-30 12:55:43,688::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1752::DEBUG::2012-08-30 12:55:43,688::task::978::TaskManager.Task::(_decref) Task=`88738fb9-06f6-4fef-9d6f-40b1b82994e2`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:55:45,226::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:45,226::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1753::DEBUG::2012-08-30 12:55:46,883::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1753::DEBUG::2012-08-30 12:55:46,884::task::588::TaskManager.Task::(_updateState) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::moving from state init -> state preparing
>Thread-1753::INFO::2012-08-30 12:55:46,884::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1753::DEBUG::2012-08-30 12:55:46,885::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`bfe92591-7d04-400e-8ca5-98ed1c09ae1c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1753::DEBUG::2012-08-30 12:55:46,885::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1753::DEBUG::2012-08-30 12:55:46,885::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1753::DEBUG::2012-08-30 12:55:46,886::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`bfe92591-7d04-400e-8ca5-98ed1c09ae1c`::Granted request
>Thread-1753::DEBUG::2012-08-30 12:55:46,886::task::817::TaskManager.Task::(resourceAcquired) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1753::DEBUG::2012-08-30 12:55:46,887::task::978::TaskManager.Task::(_decref) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::ref 1 aborting False
>Thread-1753::INFO::2012-08-30 12:55:46,890::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1753::DEBUG::2012-08-30 12:55:46,890::task::1172::TaskManager.Task::(prepare) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1753::DEBUG::2012-08-30 12:55:46,890::task::588::TaskManager.Task::(_updateState) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::moving from state preparing -> state finished
>Thread-1753::DEBUG::2012-08-30 12:55:46,890::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1753::DEBUG::2012-08-30 12:55:46,891::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1753::DEBUG::2012-08-30 12:55:46,891::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1753::DEBUG::2012-08-30 12:55:46,891::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1753::DEBUG::2012-08-30 12:55:46,891::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1753::DEBUG::2012-08-30 12:55:46,892::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1753::DEBUG::2012-08-30 12:55:46,892::task::978::TaskManager.Task::(_decref) Task=`21c231b5-0b49-40a4-bde9-5fe8e63aca57`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:55:48,195::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:50,227::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1755::DEBUG::2012-08-30 12:55:52,067::task::588::TaskManager.Task::(_updateState) Task=`b6131399-df4c-461b-bfc7-31098b94103f`::moving from state init -> state preparing
>Thread-1755::INFO::2012-08-30 12:55:52,067::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1755::INFO::2012-08-30 12:55:52,068::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00175499916077', 'lastCheck': 1346302549.9230909, 'code': 0, 'valid': True}}
>Thread-1755::DEBUG::2012-08-30 12:55:52,068::task::1172::TaskManager.Task::(prepare) Task=`b6131399-df4c-461b-bfc7-31098b94103f`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00175499916077', 'lastCheck': 1346302549.9230909, 'code': 0, 'valid': True}}
>Thread-1755::DEBUG::2012-08-30 12:55:52,068::task::588::TaskManager.Task::(_updateState) Task=`b6131399-df4c-461b-bfc7-31098b94103f`::moving from state preparing -> state finished
>Thread-1755::DEBUG::2012-08-30 12:55:52,068::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1755::DEBUG::2012-08-30 12:55:52,068::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1755::DEBUG::2012-08-30 12:55:52,069::task::978::TaskManager.Task::(_decref) Task=`b6131399-df4c-461b-bfc7-31098b94103f`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 12:55:53,196::task::588::TaskManager.Task::(_updateState) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:55:53,197::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:55:53,197::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9c1c1f73-e56d-41fe-b7f9-36096cf3fa2d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:55:53,197::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:55:53,197::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:55:53,198::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9c1c1f73-e56d-41fe-b7f9-36096cf3fa2d`::Granted request
>Thread-40::DEBUG::2012-08-30 12:55:53,198::task::817::TaskManager.Task::(resourceAcquired) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:55:53,198::task::978::TaskManager.Task::(_decref) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:55:53,200::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:55:53,202::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:55:53,203::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:55:53,203::task::1172::TaskManager.Task::(prepare) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:55:53,204::task::588::TaskManager.Task::(_updateState) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:55:53,204::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:55:53,204::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:55:53,204::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:55:53,205::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:55:53,205::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:55:53,205::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:55:53,205::task::978::TaskManager.Task::(_decref) Task=`535f3da1-0d32-4abe-9462-181b6f73e2d0`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:55:53,206::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:53,206::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:53,207::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:55:53,207::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:55:55,229::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1756::DEBUG::2012-08-30 12:55:55,278::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1756::DEBUG::2012-08-30 12:55:55,278::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>VM Channels Listener::DEBUG::2012-08-30 12:55:55,322::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 12:55:58,208::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:00,230::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:00,231::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1757::DEBUG::2012-08-30 12:56:00,358::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1757::DEBUG::2012-08-30 12:56:00,359::task::588::TaskManager.Task::(_updateState) Task=`b20025d5-6ccd-4f15-baec-a38e287d30d4`::moving from state init -> state preparing
>Thread-1757::INFO::2012-08-30 12:56:00,359::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1757::INFO::2012-08-30 12:56:00,359::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1757::DEBUG::2012-08-30 12:56:00,360::task::1172::TaskManager.Task::(prepare) Task=`b20025d5-6ccd-4f15-baec-a38e287d30d4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1757::DEBUG::2012-08-30 12:56:00,360::task::588::TaskManager.Task::(_updateState) Task=`b20025d5-6ccd-4f15-baec-a38e287d30d4`::moving from state preparing -> state finished
>Thread-1757::DEBUG::2012-08-30 12:56:00,360::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1757::DEBUG::2012-08-30 12:56:00,361::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1757::DEBUG::2012-08-30 12:56:00,361::task::978::TaskManager.Task::(_decref) Task=`b20025d5-6ccd-4f15-baec-a38e287d30d4`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 12:56:02,329::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:56:03,209::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1759::DEBUG::2012-08-30 12:56:03,562::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1759::DEBUG::2012-08-30 12:56:03,563::task::588::TaskManager.Task::(_updateState) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::moving from state init -> state preparing
>Thread-1759::INFO::2012-08-30 12:56:03,563::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1759::DEBUG::2012-08-30 12:56:03,563::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`86f06304-3a57-44bb-ae4c-cf6961fe460a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1759::DEBUG::2012-08-30 12:56:03,564::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1759::DEBUG::2012-08-30 12:56:03,564::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1759::DEBUG::2012-08-30 12:56:03,564::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`86f06304-3a57-44bb-ae4c-cf6961fe460a`::Granted request
>Thread-1759::DEBUG::2012-08-30 12:56:03,565::task::817::TaskManager.Task::(resourceAcquired) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1759::DEBUG::2012-08-30 12:56:03,565::task::978::TaskManager.Task::(_decref) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::ref 1 aborting False
>Thread-1759::INFO::2012-08-30 12:56:03,568::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1759::DEBUG::2012-08-30 12:56:03,568::task::1172::TaskManager.Task::(prepare) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1759::DEBUG::2012-08-30 12:56:03,568::task::588::TaskManager.Task::(_updateState) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::moving from state preparing -> state finished
>Thread-1759::DEBUG::2012-08-30 12:56:03,568::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1759::DEBUG::2012-08-30 12:56:03,569::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1759::DEBUG::2012-08-30 12:56:03,569::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1759::DEBUG::2012-08-30 12:56:03,569::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1759::DEBUG::2012-08-30 12:56:03,569::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1759::DEBUG::2012-08-30 12:56:03,570::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1759::DEBUG::2012-08-30 12:56:03,570::task::978::TaskManager.Task::(_decref) Task=`a98e73e7-da55-4615-94b0-202cfd2d0382`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:56:05,232::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:08,210::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:08,210::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:10,233::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:13,211::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:15,234::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:15,235::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1763::DEBUG::2012-08-30 12:56:16,858::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1763::DEBUG::2012-08-30 12:56:16,859::task::588::TaskManager.Task::(_updateState) Task=`7301efec-f069-4ccc-b293-77766d2e32bd`::moving from state init -> state preparing
>Thread-1763::INFO::2012-08-30 12:56:16,859::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1763::INFO::2012-08-30 12:56:16,859::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1763::DEBUG::2012-08-30 12:56:16,860::task::1172::TaskManager.Task::(prepare) Task=`7301efec-f069-4ccc-b293-77766d2e32bd`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1763::DEBUG::2012-08-30 12:56:16,860::task::588::TaskManager.Task::(_updateState) Task=`7301efec-f069-4ccc-b293-77766d2e32bd`::moving from state preparing -> state finished
>Thread-1763::DEBUG::2012-08-30 12:56:16,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1763::DEBUG::2012-08-30 12:56:16,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1763::DEBUG::2012-08-30 12:56:16,861::task::978::TaskManager.Task::(_decref) Task=`7301efec-f069-4ccc-b293-77766d2e32bd`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:56:18,212::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1764::DEBUG::2012-08-30 12:56:20,060::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1765::DEBUG::2012-08-30 12:56:20,060::task::588::TaskManager.Task::(_updateState) Task=`68c756e7-b4f5-427d-87c3-2f3058216350`::moving from state init -> state preparing
>Thread-1764::DEBUG::2012-08-30 12:56:20,060::task::588::TaskManager.Task::(_updateState) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::moving from state init -> state preparing
>Thread-1765::INFO::2012-08-30 12:56:20,060::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1764::INFO::2012-08-30 12:56:20,061::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1765::INFO::2012-08-30 12:56:20,061::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00160503387451', 'lastCheck': 1346302579.9429951, 'code': 0, 'valid': True}}
>Thread-1764::DEBUG::2012-08-30 12:56:20,061::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`23a0ff0f-a409-4894-be5e-260639ef9ca7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1765::DEBUG::2012-08-30 12:56:20,061::task::1172::TaskManager.Task::(prepare) Task=`68c756e7-b4f5-427d-87c3-2f3058216350`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00160503387451', 'lastCheck': 1346302579.9429951, 'code': 0, 'valid': True}}
>Thread-1764::DEBUG::2012-08-30 12:56:20,062::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1765::DEBUG::2012-08-30 12:56:20,062::task::588::TaskManager.Task::(_updateState) Task=`68c756e7-b4f5-427d-87c3-2f3058216350`::moving from state preparing -> state finished
>Thread-1764::DEBUG::2012-08-30 12:56:20,062::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1765::DEBUG::2012-08-30 12:56:20,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1764::DEBUG::2012-08-30 12:56:20,062::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`23a0ff0f-a409-4894-be5e-260639ef9ca7`::Granted request
>Thread-1765::DEBUG::2012-08-30 12:56:20,063::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1764::DEBUG::2012-08-30 12:56:20,063::task::817::TaskManager.Task::(resourceAcquired) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1765::DEBUG::2012-08-30 12:56:20,063::task::978::TaskManager.Task::(_decref) Task=`68c756e7-b4f5-427d-87c3-2f3058216350`::ref 0 aborting False
>Thread-1764::DEBUG::2012-08-30 12:56:20,063::task::978::TaskManager.Task::(_decref) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::ref 1 aborting False
>Thread-1764::INFO::2012-08-30 12:56:20,068::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1764::DEBUG::2012-08-30 12:56:20,068::task::1172::TaskManager.Task::(prepare) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1764::DEBUG::2012-08-30 12:56:20,069::task::588::TaskManager.Task::(_updateState) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::moving from state preparing -> state finished
>Thread-1764::DEBUG::2012-08-30 12:56:20,069::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1764::DEBUG::2012-08-30 12:56:20,069::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1764::DEBUG::2012-08-30 12:56:20,070::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1764::DEBUG::2012-08-30 12:56:20,070::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1764::DEBUG::2012-08-30 12:56:20,070::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1764::DEBUG::2012-08-30 12:56:20,070::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1764::DEBUG::2012-08-30 12:56:20,070::task::978::TaskManager.Task::(_decref) Task=`510fe132-c4c9-4d58-b068-f86a41fa68e2`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:56:20,235::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:23,213::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:23,214::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1766::DEBUG::2012-08-30 12:56:23,523::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1766::DEBUG::2012-08-30 12:56:23,523::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:56:25,237::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:56:25,346::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 12:56:28,214::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:56:30,238::task::588::TaskManager.Task::(_updateState) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:56:30,238::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:56:30,238::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e39dceab-9c0e-43e6-9abc-cfe3900702b0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:56:30,239::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:56:30,239::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:56:30,239::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e39dceab-9c0e-43e6-9abc-cfe3900702b0`::Granted request
>Thread-104::DEBUG::2012-08-30 12:56:30,239::task::817::TaskManager.Task::(resourceAcquired) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:56:30,240::task::978::TaskManager.Task::(_decref) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:56:30,241::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:56:30,243::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:56:30,244::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:56:30,245::task::1172::TaskManager.Task::(prepare) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:56:30,245::task::588::TaskManager.Task::(_updateState) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:56:30,245::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:56:30,245::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:56:30,246::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:56:30,246::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:56:30,246::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:56:30,246::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:56:30,246::task::978::TaskManager.Task::(_decref) Task=`8c5a1bf5-cfc0-4cfb-b10c-8e645d916317`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:56:30,247::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:30,247::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:30,248::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:30,248::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:56:32,348::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:56:33,215::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:35,249::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1768::DEBUG::2012-08-30 12:56:35,293::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1768::DEBUG::2012-08-30 12:56:35,294::task::588::TaskManager.Task::(_updateState) Task=`d7e30343-86b5-4ca2-bb19-ac46babb8ae9`::moving from state init -> state preparing
>Thread-1768::INFO::2012-08-30 12:56:35,295::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1768::INFO::2012-08-30 12:56:35,295::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1768::DEBUG::2012-08-30 12:56:35,295::task::1172::TaskManager.Task::(prepare) Task=`d7e30343-86b5-4ca2-bb19-ac46babb8ae9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1768::DEBUG::2012-08-30 12:56:35,296::task::588::TaskManager.Task::(_updateState) Task=`d7e30343-86b5-4ca2-bb19-ac46babb8ae9`::moving from state preparing -> state finished
>Thread-1768::DEBUG::2012-08-30 12:56:35,296::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1768::DEBUG::2012-08-30 12:56:35,296::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1768::DEBUG::2012-08-30 12:56:35,296::task::978::TaskManager.Task::(_decref) Task=`d7e30343-86b5-4ca2-bb19-ac46babb8ae9`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:56:38,216::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:38,217::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1770::DEBUG::2012-08-30 12:56:38,493::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1770::DEBUG::2012-08-30 12:56:38,494::task::588::TaskManager.Task::(_updateState) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::moving from state init -> state preparing
>Thread-1770::INFO::2012-08-30 12:56:38,494::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1770::DEBUG::2012-08-30 12:56:38,495::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c268c2ef-27b6-49fb-9d58-8edf19e4cd46`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1770::DEBUG::2012-08-30 12:56:38,495::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1770::DEBUG::2012-08-30 12:56:38,495::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1770::DEBUG::2012-08-30 12:56:38,495::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c268c2ef-27b6-49fb-9d58-8edf19e4cd46`::Granted request
>Thread-1770::DEBUG::2012-08-30 12:56:38,496::task::817::TaskManager.Task::(resourceAcquired) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1770::DEBUG::2012-08-30 12:56:38,496::task::978::TaskManager.Task::(_decref) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::ref 1 aborting False
>Thread-1770::INFO::2012-08-30 12:56:38,499::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1770::DEBUG::2012-08-30 12:56:38,499::task::1172::TaskManager.Task::(prepare) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1770::DEBUG::2012-08-30 12:56:38,499::task::588::TaskManager.Task::(_updateState) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::moving from state preparing -> state finished
>Thread-1770::DEBUG::2012-08-30 12:56:38,499::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1770::DEBUG::2012-08-30 12:56:38,500::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1770::DEBUG::2012-08-30 12:56:38,500::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1770::DEBUG::2012-08-30 12:56:38,500::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1770::DEBUG::2012-08-30 12:56:38,500::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1770::DEBUG::2012-08-30 12:56:38,501::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1770::DEBUG::2012-08-30 12:56:38,501::task::978::TaskManager.Task::(_decref) Task=`05ddd3dd-2b9e-4767-bcab-8d736f9ffe8f`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:56:40,250::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:43,218::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:45,252::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:56:45,252::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:48,219::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1773::DEBUG::2012-08-30 12:56:50,166::task::588::TaskManager.Task::(_updateState) Task=`57a3bbd6-73fb-429e-baf9-83d9fe078880`::moving from state init -> state preparing
>Thread-1773::INFO::2012-08-30 12:56:50,166::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1773::INFO::2012-08-30 12:56:50,167::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176787376404', 'lastCheck': 1346302609.962445, 'code': 0, 'valid': True}}
>Thread-1773::DEBUG::2012-08-30 12:56:50,167::task::1172::TaskManager.Task::(prepare) Task=`57a3bbd6-73fb-429e-baf9-83d9fe078880`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176787376404', 'lastCheck': 1346302609.962445, 'code': 0, 'valid': True}}
>Thread-1773::DEBUG::2012-08-30 12:56:50,167::task::588::TaskManager.Task::(_updateState) Task=`57a3bbd6-73fb-429e-baf9-83d9fe078880`::moving from state preparing -> state finished
>Thread-1773::DEBUG::2012-08-30 12:56:50,167::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1773::DEBUG::2012-08-30 12:56:50,168::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1773::DEBUG::2012-08-30 12:56:50,168::task::978::TaskManager.Task::(_decref) Task=`57a3bbd6-73fb-429e-baf9-83d9fe078880`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:56:50,254::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1774::DEBUG::2012-08-30 12:56:51,760::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1774::DEBUG::2012-08-30 12:56:51,761::task::588::TaskManager.Task::(_updateState) Task=`f2756b39-7b44-4b46-ae42-4eed386cc7b0`::moving from state init -> state preparing
>Thread-1774::INFO::2012-08-30 12:56:51,761::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1774::INFO::2012-08-30 12:56:51,761::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1774::DEBUG::2012-08-30 12:56:51,762::task::1172::TaskManager.Task::(prepare) Task=`f2756b39-7b44-4b46-ae42-4eed386cc7b0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1774::DEBUG::2012-08-30 12:56:51,762::task::588::TaskManager.Task::(_updateState) Task=`f2756b39-7b44-4b46-ae42-4eed386cc7b0`::moving from state preparing -> state finished
>Thread-1774::DEBUG::2012-08-30 12:56:51,762::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1774::DEBUG::2012-08-30 12:56:51,762::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1774::DEBUG::2012-08-30 12:56:51,763::task::978::TaskManager.Task::(_decref) Task=`f2756b39-7b44-4b46-ae42-4eed386cc7b0`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 12:56:53,220::task::588::TaskManager.Task::(_updateState) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:56:53,220::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:56:53,221::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`477cd298-0633-46b9-b114-8d4a4e5a2920`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:56:53,221::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:56:53,221::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:56:53,221::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`477cd298-0633-46b9-b114-8d4a4e5a2920`::Granted request
>Thread-40::DEBUG::2012-08-30 12:56:53,222::task::817::TaskManager.Task::(resourceAcquired) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:56:53,222::task::978::TaskManager.Task::(_decref) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:56:53,224::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:56:53,226::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:56:53,227::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:56:53,227::task::1172::TaskManager.Task::(prepare) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:56:53,227::task::588::TaskManager.Task::(_updateState) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:56:53,228::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:56:53,228::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:56:53,228::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:56:53,228::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:56:53,229::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:56:53,229::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:56:53,229::task::978::TaskManager.Task::(_decref) Task=`9192fce4-c429-4cee-86bd-9ea4f96dc467`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:56:53,229::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:53,230::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:53,230::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:56:53,231::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1775::DEBUG::2012-08-30 12:56:53,350::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1775::DEBUG::2012-08-30 12:56:53,350::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1776::DEBUG::2012-08-30 12:56:54,940::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1776::DEBUG::2012-08-30 12:56:54,940::task::588::TaskManager.Task::(_updateState) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::moving from state init -> state preparing
>Thread-1776::INFO::2012-08-30 12:56:54,940::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1776::DEBUG::2012-08-30 12:56:54,941::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a011c9e7-4d3f-4e45-95aa-2a0059c11a89`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1776::DEBUG::2012-08-30 12:56:54,941::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1776::DEBUG::2012-08-30 12:56:54,941::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1776::DEBUG::2012-08-30 12:56:54,941::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a011c9e7-4d3f-4e45-95aa-2a0059c11a89`::Granted request
>Thread-1776::DEBUG::2012-08-30 12:56:54,942::task::817::TaskManager.Task::(resourceAcquired) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1776::DEBUG::2012-08-30 12:56:54,942::task::978::TaskManager.Task::(_decref) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::ref 1 aborting False
>Thread-1776::INFO::2012-08-30 12:56:54,945::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1776::DEBUG::2012-08-30 12:56:54,945::task::1172::TaskManager.Task::(prepare) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1776::DEBUG::2012-08-30 12:56:54,945::task::588::TaskManager.Task::(_updateState) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::moving from state preparing -> state finished
>Thread-1776::DEBUG::2012-08-30 12:56:54,946::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1776::DEBUG::2012-08-30 12:56:54,946::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1776::DEBUG::2012-08-30 12:56:54,946::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1776::DEBUG::2012-08-30 12:56:54,946::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1776::DEBUG::2012-08-30 12:56:54,947::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1776::DEBUG::2012-08-30 12:56:54,947::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1776::DEBUG::2012-08-30 12:56:54,947::task::978::TaskManager.Task::(_decref) Task=`8dfe6f71-0618-4465-9f90-f360d62bf96e`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:56:55,255::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:56:55,369::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 12:56:58,232::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:00,256::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:00,257::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:57:02,375::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:57:03,233::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:05,258::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:08,233::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:08,234::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1779::DEBUG::2012-08-30 12:57:08,239::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1779::DEBUG::2012-08-30 12:57:08,240::task::588::TaskManager.Task::(_updateState) Task=`276c7be3-f971-4b66-848e-c126db0e8a7f`::moving from state init -> state preparing
>Thread-1779::INFO::2012-08-30 12:57:08,240::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1779::INFO::2012-08-30 12:57:08,240::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1779::DEBUG::2012-08-30 12:57:08,240::task::1172::TaskManager.Task::(prepare) Task=`276c7be3-f971-4b66-848e-c126db0e8a7f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1779::DEBUG::2012-08-30 12:57:08,241::task::588::TaskManager.Task::(_updateState) Task=`276c7be3-f971-4b66-848e-c126db0e8a7f`::moving from state preparing -> state finished
>Thread-1779::DEBUG::2012-08-30 12:57:08,241::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1779::DEBUG::2012-08-30 12:57:08,241::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1779::DEBUG::2012-08-30 12:57:08,241::task::978::TaskManager.Task::(_decref) Task=`276c7be3-f971-4b66-848e-c126db0e8a7f`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:57:10,259::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1781::DEBUG::2012-08-30 12:57:11,429::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1781::DEBUG::2012-08-30 12:57:11,430::task::588::TaskManager.Task::(_updateState) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::moving from state init -> state preparing
>Thread-1781::INFO::2012-08-30 12:57:11,430::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1781::DEBUG::2012-08-30 12:57:11,431::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e39b3495-dc36-45cd-91d4-f955d1ba5edb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1781::DEBUG::2012-08-30 12:57:11,431::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1781::DEBUG::2012-08-30 12:57:11,431::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1781::DEBUG::2012-08-30 12:57:11,431::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e39b3495-dc36-45cd-91d4-f955d1ba5edb`::Granted request
>Thread-1781::DEBUG::2012-08-30 12:57:11,432::task::817::TaskManager.Task::(resourceAcquired) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1781::DEBUG::2012-08-30 12:57:11,432::task::978::TaskManager.Task::(_decref) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::ref 1 aborting False
>Thread-1781::INFO::2012-08-30 12:57:11,435::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1781::DEBUG::2012-08-30 12:57:11,435::task::1172::TaskManager.Task::(prepare) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1781::DEBUG::2012-08-30 12:57:11,435::task::588::TaskManager.Task::(_updateState) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::moving from state preparing -> state finished
>Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1781::DEBUG::2012-08-30 12:57:11,436::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1781::DEBUG::2012-08-30 12:57:11,437::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1781::DEBUG::2012-08-30 12:57:11,437::task::978::TaskManager.Task::(_decref) Task=`8c4cd191-832c-4d8e-a02c-6d0cbcc82b8b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:57:13,235::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:15,260::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:15,261::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:18,236::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1783::DEBUG::2012-08-30 12:57:19,597::task::588::TaskManager.Task::(_updateState) Task=`868ee6c2-5800-413a-ad57-fa5c91732c90`::moving from state init -> state preparing
>Thread-1783::INFO::2012-08-30 12:57:19,598::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1783::INFO::2012-08-30 12:57:19,598::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346302629.976028, 'code': 0, 'valid': True}}
>Thread-1783::DEBUG::2012-08-30 12:57:19,598::task::1172::TaskManager.Task::(prepare) Task=`868ee6c2-5800-413a-ad57-fa5c91732c90`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346302629.976028, 'code': 0, 'valid': True}}
>Thread-1783::DEBUG::2012-08-30 12:57:19,599::task::588::TaskManager.Task::(_updateState) Task=`868ee6c2-5800-413a-ad57-fa5c91732c90`::moving from state preparing -> state finished
>Thread-1783::DEBUG::2012-08-30 12:57:19,599::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1783::DEBUG::2012-08-30 12:57:19,599::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1783::DEBUG::2012-08-30 12:57:19,599::task::978::TaskManager.Task::(_decref) Task=`868ee6c2-5800-413a-ad57-fa5c91732c90`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:57:20,262::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1784::DEBUG::2012-08-30 12:57:22,815::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1784::DEBUG::2012-08-30 12:57:22,815::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 12:57:23,237::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:23,237::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1785::DEBUG::2012-08-30 12:57:24,400::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1785::DEBUG::2012-08-30 12:57:24,401::task::588::TaskManager.Task::(_updateState) Task=`d12a60ad-197b-4576-9466-9d872315d6de`::moving from state init -> state preparing
>Thread-1785::INFO::2012-08-30 12:57:24,401::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1785::INFO::2012-08-30 12:57:24,401::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1785::DEBUG::2012-08-30 12:57:24,401::task::1172::TaskManager.Task::(prepare) Task=`d12a60ad-197b-4576-9466-9d872315d6de`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1785::DEBUG::2012-08-30 12:57:24,402::task::588::TaskManager.Task::(_updateState) Task=`d12a60ad-197b-4576-9466-9d872315d6de`::moving from state preparing -> state finished
>Thread-1785::DEBUG::2012-08-30 12:57:24,402::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1785::DEBUG::2012-08-30 12:57:24,402::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1785::DEBUG::2012-08-30 12:57:24,402::task::978::TaskManager.Task::(_decref) Task=`d12a60ad-197b-4576-9466-9d872315d6de`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:57:25,264::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:57:25,393::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1786::DEBUG::2012-08-30 12:57:27,604::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1786::DEBUG::2012-08-30 12:57:27,605::task::588::TaskManager.Task::(_updateState) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::moving from state init -> state preparing
>Thread-1786::INFO::2012-08-30 12:57:27,605::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1786::DEBUG::2012-08-30 12:57:27,606::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`144b02af-5759-4605-9587-f791b0e92068`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1786::DEBUG::2012-08-30 12:57:27,606::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1786::DEBUG::2012-08-30 12:57:27,606::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1786::DEBUG::2012-08-30 12:57:27,607::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`144b02af-5759-4605-9587-f791b0e92068`::Granted request
>Thread-1786::DEBUG::2012-08-30 12:57:27,607::task::817::TaskManager.Task::(resourceAcquired) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1786::DEBUG::2012-08-30 12:57:27,607::task::978::TaskManager.Task::(_decref) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::ref 1 aborting False
>Thread-1786::INFO::2012-08-30 12:57:27,610::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1786::DEBUG::2012-08-30 12:57:27,610::task::1172::TaskManager.Task::(prepare) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1786::DEBUG::2012-08-30 12:57:27,611::task::588::TaskManager.Task::(_updateState) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::moving from state preparing -> state finished
>Thread-1786::DEBUG::2012-08-30 12:57:27,611::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1786::DEBUG::2012-08-30 12:57:27,611::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1786::DEBUG::2012-08-30 12:57:27,611::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1786::DEBUG::2012-08-30 12:57:27,612::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1786::DEBUG::2012-08-30 12:57:27,612::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1786::DEBUG::2012-08-30 12:57:27,612::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1786::DEBUG::2012-08-30 12:57:27,612::task::978::TaskManager.Task::(_decref) Task=`e698cfc8-0d61-482a-ad93-b744e315ad8f`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:57:28,238::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:57:30,265::task::588::TaskManager.Task::(_updateState) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:57:30,265::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:57:30,265::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c0c44eef-913f-44ee-977d-85e337da99f0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:57:30,266::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:57:30,266::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:57:30,266::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c0c44eef-913f-44ee-977d-85e337da99f0`::Granted request
>Thread-104::DEBUG::2012-08-30 12:57:30,267::task::817::TaskManager.Task::(resourceAcquired) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:57:30,267::task::978::TaskManager.Task::(_decref) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:57:30,268::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:57:30,270::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:57:30,271::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:57:30,272::task::1172::TaskManager.Task::(prepare) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:57:30,272::task::588::TaskManager.Task::(_updateState) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:57:30,272::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:57:30,272::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:57:30,273::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:57:30,273::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:57:30,273::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:57:30,273::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:57:30,274::task::978::TaskManager.Task::(_decref) Task=`59d23d3d-03a5-4107-aa53-46ac20871ce5`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:57:30,274::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:30,274::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:30,275::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:30,275::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:57:32,400::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:57:33,239::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:35,276::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:38,240::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:38,241::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:40,278::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1790::DEBUG::2012-08-30 12:57:40,834::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1790::DEBUG::2012-08-30 12:57:40,834::task::588::TaskManager.Task::(_updateState) Task=`4f5f6d99-52b0-4dce-b402-e4e111883cb9`::moving from state init -> state preparing
>Thread-1790::INFO::2012-08-30 12:57:40,835::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1790::INFO::2012-08-30 12:57:40,835::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1790::DEBUG::2012-08-30 12:57:40,835::task::1172::TaskManager.Task::(prepare) Task=`4f5f6d99-52b0-4dce-b402-e4e111883cb9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1790::DEBUG::2012-08-30 12:57:40,835::task::588::TaskManager.Task::(_updateState) Task=`4f5f6d99-52b0-4dce-b402-e4e111883cb9`::moving from state preparing -> state finished
>Thread-1790::DEBUG::2012-08-30 12:57:40,836::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1790::DEBUG::2012-08-30 12:57:40,836::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1790::DEBUG::2012-08-30 12:57:40,836::task::978::TaskManager.Task::(_decref) Task=`4f5f6d99-52b0-4dce-b402-e4e111883cb9`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:57:43,242::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1792::DEBUG::2012-08-30 12:57:44,021::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1792::DEBUG::2012-08-30 12:57:44,021::task::588::TaskManager.Task::(_updateState) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::moving from state init -> state preparing
>Thread-1792::INFO::2012-08-30 12:57:44,021::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1792::DEBUG::2012-08-30 12:57:44,022::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a9ff7bba-a9c1-4d7f-8177-0b6ebdc53094`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1792::DEBUG::2012-08-30 12:57:44,022::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1792::DEBUG::2012-08-30 12:57:44,022::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1792::DEBUG::2012-08-30 12:57:44,023::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a9ff7bba-a9c1-4d7f-8177-0b6ebdc53094`::Granted request
>Thread-1792::DEBUG::2012-08-30 12:57:44,023::task::817::TaskManager.Task::(resourceAcquired) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1792::DEBUG::2012-08-30 12:57:44,023::task::978::TaskManager.Task::(_decref) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::ref 1 aborting False
>Thread-1792::INFO::2012-08-30 12:57:44,026::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1792::DEBUG::2012-08-30 12:57:44,026::task::1172::TaskManager.Task::(prepare) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1792::DEBUG::2012-08-30 12:57:44,027::task::588::TaskManager.Task::(_updateState) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::moving from state preparing -> state finished
>Thread-1792::DEBUG::2012-08-30 12:57:44,027::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1792::DEBUG::2012-08-30 12:57:44,027::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1792::DEBUG::2012-08-30 12:57:44,027::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1792::DEBUG::2012-08-30 12:57:44,028::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1792::DEBUG::2012-08-30 12:57:44,028::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1792::DEBUG::2012-08-30 12:57:44,028::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1792::DEBUG::2012-08-30 12:57:44,028::task::978::TaskManager.Task::(_decref) Task=`58ecd850-79d1-4490-89d6-6e916903aa61`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:57:45,279::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:45,279::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:48,242::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1793::DEBUG::2012-08-30 12:57:48,392::task::588::TaskManager.Task::(_updateState) Task=`ae121905-b38d-4412-bf49-ac4481e00b60`::moving from state init -> state preparing
>Thread-1793::INFO::2012-08-30 12:57:48,392::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1793::INFO::2012-08-30 12:57:48,392::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346302659.9950261, 'code': 0, 'valid': True}}
>Thread-1793::DEBUG::2012-08-30 12:57:48,392::task::1172::TaskManager.Task::(prepare) Task=`ae121905-b38d-4412-bf49-ac4481e00b60`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346302659.9950261, 'code': 0, 'valid': True}}
>Thread-1793::DEBUG::2012-08-30 12:57:48,393::task::588::TaskManager.Task::(_updateState) Task=`ae121905-b38d-4412-bf49-ac4481e00b60`::moving from state preparing -> state finished
>Thread-1793::DEBUG::2012-08-30 12:57:48,393::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1793::DEBUG::2012-08-30 12:57:48,393::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1793::DEBUG::2012-08-30 12:57:48,393::task::978::TaskManager.Task::(_decref) Task=`ae121905-b38d-4412-bf49-ac4481e00b60`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:57:50,281::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1794::DEBUG::2012-08-30 12:57:51,579::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1794::DEBUG::2012-08-30 12:57:51,580::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::DEBUG::2012-08-30 12:57:53,243::task::588::TaskManager.Task::(_updateState) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:57:53,244::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:57:53,244::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`1caa370f-00d7-4111-a6f9-77bb9b0ccee1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:57:53,244::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:57:53,245::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:57:53,245::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`1caa370f-00d7-4111-a6f9-77bb9b0ccee1`::Granted request
>Thread-40::DEBUG::2012-08-30 12:57:53,245::task::817::TaskManager.Task::(resourceAcquired) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:57:53,245::task::978::TaskManager.Task::(_decref) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:57:53,247::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:57:53,249::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:57:53,250::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:57:53,251::task::1172::TaskManager.Task::(prepare) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:57:53,251::task::588::TaskManager.Task::(_updateState) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:57:53,251::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:57:53,251::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:57:53,252::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:57:53,252::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:57:53,252::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:57:53,252::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:57:53,253::task::978::TaskManager.Task::(_decref) Task=`de4e5b94-ae11-4473-b14a-5a9bec0b17dd`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:57:53,253::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:53,253::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:53,254::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:57:53,254::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:57:55,282::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:57:55,421::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1796::DEBUG::2012-08-30 12:57:56,843::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1796::DEBUG::2012-08-30 12:57:56,844::task::588::TaskManager.Task::(_updateState) Task=`2bf4d963-11a0-4eab-9c35-a4248fdc28b8`::moving from state init -> state preparing
>Thread-1796::INFO::2012-08-30 12:57:56,844::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1796::INFO::2012-08-30 12:57:56,844::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1796::DEBUG::2012-08-30 12:57:56,845::task::1172::TaskManager.Task::(prepare) Task=`2bf4d963-11a0-4eab-9c35-a4248fdc28b8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1796::DEBUG::2012-08-30 12:57:56,845::task::588::TaskManager.Task::(_updateState) Task=`2bf4d963-11a0-4eab-9c35-a4248fdc28b8`::moving from state preparing -> state finished
>Thread-1796::DEBUG::2012-08-30 12:57:56,845::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1796::DEBUG::2012-08-30 12:57:56,845::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1796::DEBUG::2012-08-30 12:57:56,846::task::978::TaskManager.Task::(_decref) Task=`2bf4d963-11a0-4eab-9c35-a4248fdc28b8`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:57:58,255::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1797::DEBUG::2012-08-30 12:58:00,056::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1797::DEBUG::2012-08-30 12:58:00,056::task::588::TaskManager.Task::(_updateState) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::moving from state init -> state preparing
>Thread-1797::INFO::2012-08-30 12:58:00,056::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1797::DEBUG::2012-08-30 12:58:00,057::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e31b2058-7129-4bd5-9d0d-921a8ceba4ed`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1797::DEBUG::2012-08-30 12:58:00,057::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1797::DEBUG::2012-08-30 12:58:00,057::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1797::DEBUG::2012-08-30 12:58:00,058::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e31b2058-7129-4bd5-9d0d-921a8ceba4ed`::Granted request
>Thread-1797::DEBUG::2012-08-30 12:58:00,058::task::817::TaskManager.Task::(resourceAcquired) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1797::DEBUG::2012-08-30 12:58:00,058::task::978::TaskManager.Task::(_decref) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::ref 1 aborting False
>Thread-1797::INFO::2012-08-30 12:58:00,061::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1797::DEBUG::2012-08-30 12:58:00,061::task::1172::TaskManager.Task::(prepare) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1797::DEBUG::2012-08-30 12:58:00,062::task::588::TaskManager.Task::(_updateState) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::moving from state preparing -> state finished
>Thread-1797::DEBUG::2012-08-30 12:58:00,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1797::DEBUG::2012-08-30 12:58:00,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1797::DEBUG::2012-08-30 12:58:00,062::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1797::DEBUG::2012-08-30 12:58:00,063::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1797::DEBUG::2012-08-30 12:58:00,063::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1797::DEBUG::2012-08-30 12:58:00,063::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1797::DEBUG::2012-08-30 12:58:00,063::task::978::TaskManager.Task::(_decref) Task=`93588481-2be2-4d6f-abbd-3694ebebe6d6`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:58:00,283::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:00,284::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:58:02,426::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:58:03,256::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:05,285::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:08,257::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:08,257::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:10,286::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:13,258::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1801::DEBUG::2012-08-30 12:58:14,091::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1801::DEBUG::2012-08-30 12:58:14,092::task::588::TaskManager.Task::(_updateState) Task=`35edd4b5-6c63-4968-9c0c-a6412197afe6`::moving from state init -> state preparing
>Thread-1801::INFO::2012-08-30 12:58:14,092::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1801::INFO::2012-08-30 12:58:14,092::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1801::DEBUG::2012-08-30 12:58:14,092::task::1172::TaskManager.Task::(prepare) Task=`35edd4b5-6c63-4968-9c0c-a6412197afe6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1801::DEBUG::2012-08-30 12:58:14,093::task::588::TaskManager.Task::(_updateState) Task=`35edd4b5-6c63-4968-9c0c-a6412197afe6`::moving from state preparing -> state finished
>Thread-1801::DEBUG::2012-08-30 12:58:14,093::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1801::DEBUG::2012-08-30 12:58:14,093::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1801::DEBUG::2012-08-30 12:58:14,093::task::978::TaskManager.Task::(_decref) Task=`35edd4b5-6c63-4968-9c0c-a6412197afe6`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:58:15,287::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:15,288::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1802::DEBUG::2012-08-30 12:58:17,305::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1802::DEBUG::2012-08-30 12:58:17,305::task::588::TaskManager.Task::(_updateState) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::moving from state init -> state preparing
>Thread-1802::INFO::2012-08-30 12:58:17,305::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1802::DEBUG::2012-08-30 12:58:17,306::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`955d82f5-8155-424d-bce7-b2bb3772af5d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1802::DEBUG::2012-08-30 12:58:17,306::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1802::DEBUG::2012-08-30 12:58:17,306::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1802::DEBUG::2012-08-30 12:58:17,306::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`955d82f5-8155-424d-bce7-b2bb3772af5d`::Granted request
>Thread-1802::DEBUG::2012-08-30 12:58:17,307::task::817::TaskManager.Task::(resourceAcquired) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1802::DEBUG::2012-08-30 12:58:17,307::task::978::TaskManager.Task::(_decref) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::ref 1 aborting False
>Thread-1802::INFO::2012-08-30 12:58:17,310::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1802::DEBUG::2012-08-30 12:58:17,310::task::1172::TaskManager.Task::(prepare) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1802::DEBUG::2012-08-30 12:58:17,310::task::588::TaskManager.Task::(_updateState) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::moving from state preparing -> state finished
>Thread-1802::DEBUG::2012-08-30 12:58:17,311::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1802::DEBUG::2012-08-30 12:58:17,311::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1802::DEBUG::2012-08-30 12:58:17,311::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1802::DEBUG::2012-08-30 12:58:17,311::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1802::DEBUG::2012-08-30 12:58:17,312::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1802::DEBUG::2012-08-30 12:58:17,312::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1802::DEBUG::2012-08-30 12:58:17,312::task::978::TaskManager.Task::(_decref) Task=`64104e87-993e-4d17-825b-bbd5d7f4ab6f`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:58:18,259::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1803::DEBUG::2012-08-30 12:58:18,918::task::588::TaskManager.Task::(_updateState) Task=`bd6a93ac-948b-4bf1-911a-947fef35d8ae`::moving from state init -> state preparing
>Thread-1803::INFO::2012-08-30 12:58:18,918::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1803::INFO::2012-08-30 12:58:18,918::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346302690.0157461, 'code': 0, 'valid': True}}
>Thread-1803::DEBUG::2012-08-30 12:58:18,919::task::1172::TaskManager.Task::(prepare) Task=`bd6a93ac-948b-4bf1-911a-947fef35d8ae`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346302690.0157461, 'code': 0, 'valid': True}}
>Thread-1803::DEBUG::2012-08-30 12:58:18,919::task::588::TaskManager.Task::(_updateState) Task=`bd6a93ac-948b-4bf1-911a-947fef35d8ae`::moving from state preparing -> state finished
>Thread-1803::DEBUG::2012-08-30 12:58:18,919::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1803::DEBUG::2012-08-30 12:58:18,919::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1803::DEBUG::2012-08-30 12:58:18,920::task::978::TaskManager.Task::(_decref) Task=`bd6a93ac-948b-4bf1-911a-947fef35d8ae`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:58:20,289::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1804::DEBUG::2012-08-30 12:58:22,224::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1804::DEBUG::2012-08-30 12:58:22,225::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 12:58:23,260::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:23,261::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:25,290::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:58:25,441::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 12:58:28,262::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:58:30,291::task::588::TaskManager.Task::(_updateState) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:58:30,292::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:58:30,292::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`8afa8b36-941c-47fc-b3d6-4044438c542f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:58:30,292::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:58:30,293::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:58:30,293::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`8afa8b36-941c-47fc-b3d6-4044438c542f`::Granted request
>Thread-104::DEBUG::2012-08-30 12:58:30,293::task::817::TaskManager.Task::(resourceAcquired) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:58:30,294::task::978::TaskManager.Task::(_decref) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:58:30,295::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:58:30,297::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:58:30,298::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:58:30,298::task::1172::TaskManager.Task::(prepare) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:58:30,299::task::588::TaskManager.Task::(_updateState) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:58:30,299::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:58:30,299::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:58:30,299::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:58:30,300::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:58:30,300::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:58:30,300::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:58:30,300::task::978::TaskManager.Task::(_decref) Task=`165b11b9-c5bc-4787-bf7e-0fc3267ea681`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:58:30,301::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:30,301::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:30,302::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:30,302::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:58:32,447::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1806::DEBUG::2012-08-30 12:58:32,729::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1806::DEBUG::2012-08-30 12:58:32,729::task::588::TaskManager.Task::(_updateState) Task=`b06b735b-8adf-44a6-a7b1-1a1b377ad2b1`::moving from state init -> state preparing
>Thread-1806::INFO::2012-08-30 12:58:32,729::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1806::INFO::2012-08-30 12:58:32,730::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1806::DEBUG::2012-08-30 12:58:32,730::task::1172::TaskManager.Task::(prepare) Task=`b06b735b-8adf-44a6-a7b1-1a1b377ad2b1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1806::DEBUG::2012-08-30 12:58:32,730::task::588::TaskManager.Task::(_updateState) Task=`b06b735b-8adf-44a6-a7b1-1a1b377ad2b1`::moving from state preparing -> state finished
>Thread-1806::DEBUG::2012-08-30 12:58:32,730::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1806::DEBUG::2012-08-30 12:58:32,731::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1806::DEBUG::2012-08-30 12:58:32,731::task::978::TaskManager.Task::(_decref) Task=`b06b735b-8adf-44a6-a7b1-1a1b377ad2b1`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:58:33,262::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:35,303::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1808::DEBUG::2012-08-30 12:58:36,039::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1808::DEBUG::2012-08-30 12:58:36,039::task::588::TaskManager.Task::(_updateState) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::moving from state init -> state preparing
>Thread-1808::INFO::2012-08-30 12:58:36,039::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1808::DEBUG::2012-08-30 12:58:36,040::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ff7c745a-8243-4ee5-9ae6-6c338fff913d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1808::DEBUG::2012-08-30 12:58:36,040::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1808::DEBUG::2012-08-30 12:58:36,040::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1808::DEBUG::2012-08-30 12:58:36,040::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ff7c745a-8243-4ee5-9ae6-6c338fff913d`::Granted request
>Thread-1808::DEBUG::2012-08-30 12:58:36,041::task::817::TaskManager.Task::(resourceAcquired) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1808::DEBUG::2012-08-30 12:58:36,041::task::978::TaskManager.Task::(_decref) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::ref 1 aborting False
>Thread-1808::INFO::2012-08-30 12:58:36,044::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1808::DEBUG::2012-08-30 12:58:36,044::task::1172::TaskManager.Task::(prepare) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1808::DEBUG::2012-08-30 12:58:36,045::task::588::TaskManager.Task::(_updateState) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::moving from state preparing -> state finished
>Thread-1808::DEBUG::2012-08-30 12:58:36,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1808::DEBUG::2012-08-30 12:58:36,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1808::DEBUG::2012-08-30 12:58:36,045::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1808::DEBUG::2012-08-30 12:58:36,045::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1808::DEBUG::2012-08-30 12:58:36,046::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1808::DEBUG::2012-08-30 12:58:36,046::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1808::DEBUG::2012-08-30 12:58:36,046::task::978::TaskManager.Task::(_decref) Task=`bccc461c-eca9-421d-89ef-39fc7c5157c5`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:58:38,263::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:38,264::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:40,304::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:43,265::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:45,306::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:58:45,306::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:48,266::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1811::DEBUG::2012-08-30 12:58:49,298::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1811::DEBUG::2012-08-30 12:58:49,299::task::588::TaskManager.Task::(_updateState) Task=`0714c63d-b461-4dd6-8d4b-1ee0599a14ce`::moving from state init -> state preparing
>Thread-1811::INFO::2012-08-30 12:58:49,299::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1811::INFO::2012-08-30 12:58:49,299::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1811::DEBUG::2012-08-30 12:58:49,300::task::1172::TaskManager.Task::(prepare) Task=`0714c63d-b461-4dd6-8d4b-1ee0599a14ce`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1811::DEBUG::2012-08-30 12:58:49,300::task::588::TaskManager.Task::(_updateState) Task=`0714c63d-b461-4dd6-8d4b-1ee0599a14ce`::moving from state preparing -> state finished
>Thread-1811::DEBUG::2012-08-30 12:58:49,300::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1811::DEBUG::2012-08-30 12:58:49,300::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1811::DEBUG::2012-08-30 12:58:49,301::task::978::TaskManager.Task::(_decref) Task=`0714c63d-b461-4dd6-8d4b-1ee0599a14ce`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:58:50,308::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1812::DEBUG::2012-08-30 12:58:50,317::task::588::TaskManager.Task::(_updateState) Task=`2d711989-8508-45a0-be85-8648f76f13fd`::moving from state init -> state preparing
>Thread-1812::INFO::2012-08-30 12:58:50,317::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1812::INFO::2012-08-30 12:58:50,317::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176596641541', 'lastCheck': 1346302730.0416851, 'code': 0, 'valid': True}}
>Thread-1812::DEBUG::2012-08-30 12:58:50,318::task::1172::TaskManager.Task::(prepare) Task=`2d711989-8508-45a0-be85-8648f76f13fd`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176596641541', 'lastCheck': 1346302730.0416851, 'code': 0, 'valid': True}}
>Thread-1812::DEBUG::2012-08-30 12:58:50,318::task::588::TaskManager.Task::(_updateState) Task=`2d711989-8508-45a0-be85-8648f76f13fd`::moving from state preparing -> state finished
>Thread-1812::DEBUG::2012-08-30 12:58:50,318::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1812::DEBUG::2012-08-30 12:58:50,318::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1812::DEBUG::2012-08-30 12:58:50,319::task::978::TaskManager.Task::(_decref) Task=`2d711989-8508-45a0-be85-8648f76f13fd`::ref 0 aborting False
>Thread-1813::DEBUG::2012-08-30 12:58:51,930::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1813::DEBUG::2012-08-30 12:58:51,930::task::588::TaskManager.Task::(_updateState) Task=`82697733-fdf1-49b5-9001-cf605785541e`::moving from state init -> state preparing
>Thread-1813::INFO::2012-08-30 12:58:51,930::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1813::DEBUG::2012-08-30 12:58:51,931::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6ed520ee-e823-45ff-857a-d43cd760cbef`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1813::DEBUG::2012-08-30 12:58:51,931::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1813::DEBUG::2012-08-30 12:58:51,931::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1813::DEBUG::2012-08-30 12:58:51,931::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6ed520ee-e823-45ff-857a-d43cd760cbef`::Granted request
>Thread-1813::DEBUG::2012-08-30 12:58:51,932::task::817::TaskManager.Task::(resourceAcquired) Task=`82697733-fdf1-49b5-9001-cf605785541e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1813::DEBUG::2012-08-30 12:58:51,932::task::978::TaskManager.Task::(_decref) Task=`82697733-fdf1-49b5-9001-cf605785541e`::ref 1 aborting False
>Thread-1813::INFO::2012-08-30 12:58:51,935::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1813::DEBUG::2012-08-30 12:58:51,935::task::1172::TaskManager.Task::(prepare) Task=`82697733-fdf1-49b5-9001-cf605785541e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1813::DEBUG::2012-08-30 12:58:51,935::task::588::TaskManager.Task::(_updateState) Task=`82697733-fdf1-49b5-9001-cf605785541e`::moving from state preparing -> state finished
>Thread-1813::DEBUG::2012-08-30 12:58:51,936::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1813::DEBUG::2012-08-30 12:58:51,936::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1813::DEBUG::2012-08-30 12:58:51,936::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1813::DEBUG::2012-08-30 12:58:51,936::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1813::DEBUG::2012-08-30 12:58:51,937::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1813::DEBUG::2012-08-30 12:58:51,937::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1813::DEBUG::2012-08-30 12:58:51,937::task::978::TaskManager.Task::(_decref) Task=`82697733-fdf1-49b5-9001-cf605785541e`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 12:58:53,267::task::588::TaskManager.Task::(_updateState) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:58:53,267::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:58:53,267::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bc26458a-3f7a-42dc-ba5e-60be4cb5fb6f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:58:53,268::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:58:53,268::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:58:53,268::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bc26458a-3f7a-42dc-ba5e-60be4cb5fb6f`::Granted request
>Thread-40::DEBUG::2012-08-30 12:58:53,269::task::817::TaskManager.Task::(resourceAcquired) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:58:53,269::task::978::TaskManager.Task::(_decref) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:58:53,271::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:58:53,273::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:58:53,274::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:58:53,274::task::1172::TaskManager.Task::(prepare) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:58:53,274::task::588::TaskManager.Task::(_updateState) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:58:53,274::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:58:53,275::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:58:53,275::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:58:53,275::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:58:53,275::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:58:53,276::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:58:53,276::task::978::TaskManager.Task::(_decref) Task=`1e19c708-186d-475a-bf7a-34c86ff2bd14`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:58:53,276::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:53,277::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:53,277::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:58:53,278::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1814::DEBUG::2012-08-30 12:58:53,523::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1814::DEBUG::2012-08-30 12:58:53,524::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 12:58:55,309::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:58:55,468::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 12:58:58,278::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:00,310::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:00,311::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:59:02,474::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:59:03,279::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1817::DEBUG::2012-08-30 12:59:05,162::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1817::DEBUG::2012-08-30 12:59:05,163::task::588::TaskManager.Task::(_updateState) Task=`ecc9be1d-6183-4f61-993f-615bb79036d0`::moving from state init -> state preparing
>Thread-1817::INFO::2012-08-30 12:59:05,163::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1817::INFO::2012-08-30 12:59:05,163::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1817::DEBUG::2012-08-30 12:59:05,163::task::1172::TaskManager.Task::(prepare) Task=`ecc9be1d-6183-4f61-993f-615bb79036d0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1817::DEBUG::2012-08-30 12:59:05,164::task::588::TaskManager.Task::(_updateState) Task=`ecc9be1d-6183-4f61-993f-615bb79036d0`::moving from state preparing -> state finished
>Thread-1817::DEBUG::2012-08-30 12:59:05,164::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1817::DEBUG::2012-08-30 12:59:05,164::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1817::DEBUG::2012-08-30 12:59:05,164::task::978::TaskManager.Task::(_decref) Task=`ecc9be1d-6183-4f61-993f-615bb79036d0`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:59:05,312::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1818::DEBUG::2012-08-30 12:59:08,101::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1818::DEBUG::2012-08-30 12:59:08,101::task::588::TaskManager.Task::(_updateState) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::moving from state init -> state preparing
>Thread-1818::INFO::2012-08-30 12:59:08,102::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1818::DEBUG::2012-08-30 12:59:08,102::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0f5d3e56-fdf7-437a-900f-01c2f5c8537c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1818::DEBUG::2012-08-30 12:59:08,102::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1818::DEBUG::2012-08-30 12:59:08,102::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1818::DEBUG::2012-08-30 12:59:08,103::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0f5d3e56-fdf7-437a-900f-01c2f5c8537c`::Granted request
>Thread-1818::DEBUG::2012-08-30 12:59:08,103::task::817::TaskManager.Task::(resourceAcquired) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1818::DEBUG::2012-08-30 12:59:08,103::task::978::TaskManager.Task::(_decref) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::ref 1 aborting False
>Thread-1818::INFO::2012-08-30 12:59:08,106::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1818::DEBUG::2012-08-30 12:59:08,106::task::1172::TaskManager.Task::(prepare) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1818::DEBUG::2012-08-30 12:59:08,107::task::588::TaskManager.Task::(_updateState) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::moving from state preparing -> state finished
>Thread-1818::DEBUG::2012-08-30 12:59:08,107::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1818::DEBUG::2012-08-30 12:59:08,107::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1818::DEBUG::2012-08-30 12:59:08,107::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1818::DEBUG::2012-08-30 12:59:08,108::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1818::DEBUG::2012-08-30 12:59:08,108::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1818::DEBUG::2012-08-30 12:59:08,108::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1818::DEBUG::2012-08-30 12:59:08,108::task::978::TaskManager.Task::(_decref) Task=`4bc095de-e97f-4228-bfb1-5b38e639f508`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:59:08,280::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:08,281::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:10,313::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:13,282::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:15,314::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:15,315::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:18,283::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:20,316::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1821::DEBUG::2012-08-30 12:59:20,737::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1821::DEBUG::2012-08-30 12:59:20,738::task::588::TaskManager.Task::(_updateState) Task=`8719494b-6d80-48f8-beaf-07c84be5f7b2`::moving from state init -> state preparing
>Thread-1821::INFO::2012-08-30 12:59:20,738::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1821::INFO::2012-08-30 12:59:20,738::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1821::DEBUG::2012-08-30 12:59:20,738::task::1172::TaskManager.Task::(prepare) Task=`8719494b-6d80-48f8-beaf-07c84be5f7b2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1821::DEBUG::2012-08-30 12:59:20,739::task::588::TaskManager.Task::(_updateState) Task=`8719494b-6d80-48f8-beaf-07c84be5f7b2`::moving from state preparing -> state finished
>Thread-1821::DEBUG::2012-08-30 12:59:20,739::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1821::DEBUG::2012-08-30 12:59:20,739::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1821::DEBUG::2012-08-30 12:59:20,739::task::978::TaskManager.Task::(_decref) Task=`8719494b-6d80-48f8-beaf-07c84be5f7b2`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:59:23,284::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:23,284::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1822::DEBUG::2012-08-30 12:59:24,359::task::588::TaskManager.Task::(_updateState) Task=`7f6212e2-a8ff-4354-9ad9-607c734b4f06`::moving from state init -> state preparing
>Thread-1822::INFO::2012-08-30 12:59:24,359::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1822::INFO::2012-08-30 12:59:24,360::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164389610291', 'lastCheck': 1346302760.061686, 'code': 0, 'valid': True}}
>Thread-1822::DEBUG::2012-08-30 12:59:24,360::task::1172::TaskManager.Task::(prepare) Task=`7f6212e2-a8ff-4354-9ad9-607c734b4f06`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164389610291', 'lastCheck': 1346302760.061686, 'code': 0, 'valid': True}}
>Thread-1822::DEBUG::2012-08-30 12:59:24,360::task::588::TaskManager.Task::(_updateState) Task=`7f6212e2-a8ff-4354-9ad9-607c734b4f06`::moving from state preparing -> state finished
>Thread-1822::DEBUG::2012-08-30 12:59:24,360::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1822::DEBUG::2012-08-30 12:59:24,361::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1822::DEBUG::2012-08-30 12:59:24,361::task::978::TaskManager.Task::(_decref) Task=`7f6212e2-a8ff-4354-9ad9-607c734b4f06`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:59:25,317::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:59:25,491::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1823::DEBUG::2012-08-30 12:59:26,039::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1823::DEBUG::2012-08-30 12:59:26,040::task::588::TaskManager.Task::(_updateState) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::moving from state init -> state preparing
>Thread-1823::INFO::2012-08-30 12:59:26,040::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1823::DEBUG::2012-08-30 12:59:26,040::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`4fa24500-3f49-4a18-830d-0eda438d7c80`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1823::DEBUG::2012-08-30 12:59:26,041::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1823::DEBUG::2012-08-30 12:59:26,041::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1823::DEBUG::2012-08-30 12:59:26,041::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`4fa24500-3f49-4a18-830d-0eda438d7c80`::Granted request
>Thread-1823::DEBUG::2012-08-30 12:59:26,041::task::817::TaskManager.Task::(resourceAcquired) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1823::DEBUG::2012-08-30 12:59:26,042::task::978::TaskManager.Task::(_decref) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::ref 1 aborting False
>Thread-1823::INFO::2012-08-30 12:59:26,045::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1823::DEBUG::2012-08-30 12:59:26,045::task::1172::TaskManager.Task::(prepare) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1823::DEBUG::2012-08-30 12:59:26,045::task::588::TaskManager.Task::(_updateState) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::moving from state preparing -> state finished
>Thread-1823::DEBUG::2012-08-30 12:59:26,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1823::DEBUG::2012-08-30 12:59:26,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1823::DEBUG::2012-08-30 12:59:26,046::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1823::DEBUG::2012-08-30 12:59:26,046::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1823::DEBUG::2012-08-30 12:59:26,046::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1823::DEBUG::2012-08-30 12:59:26,046::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1823::DEBUG::2012-08-30 12:59:26,047::task::978::TaskManager.Task::(_decref) Task=`c603db4f-4676-458e-ba73-e73284c0ce4a`::ref 0 aborting False
>Thread-1824::DEBUG::2012-08-30 12:59:27,631::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1824::DEBUG::2012-08-30 12:59:27,632::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 12:59:28,285::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 12:59:30,318::task::588::TaskManager.Task::(_updateState) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 12:59:30,318::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 12:59:30,319::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`693d203e-ad8b-4981-87e7-772383b37c7d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 12:59:30,319::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 12:59:30,319::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 12:59:30,320::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`693d203e-ad8b-4981-87e7-772383b37c7d`::Granted request
>Thread-104::DEBUG::2012-08-30 12:59:30,320::task::817::TaskManager.Task::(resourceAcquired) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 12:59:30,320::task::978::TaskManager.Task::(_decref) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 12:59:30,322::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 12:59:30,324::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 12:59:30,325::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:59:30,325::task::1172::TaskManager.Task::(prepare) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 12:59:30,325::task::588::TaskManager.Task::(_updateState) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 12:59:30,326::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 12:59:30,326::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 12:59:30,326::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 12:59:30,326::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 12:59:30,327::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 12:59:30,327::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 12:59:30,327::task::978::TaskManager.Task::(_decref) Task=`053d6c92-03dd-4553-b57a-8208cbc66a8f`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:59:30,327::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:30,328::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:30,328::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:30,329::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:59:32,498::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 12:59:33,286::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:35,330::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:38,287::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:38,288::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1827::DEBUG::2012-08-30 12:59:39,307::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1827::DEBUG::2012-08-30 12:59:39,307::task::588::TaskManager.Task::(_updateState) Task=`61139ffc-49c5-4e66-80f3-a1897d203d26`::moving from state init -> state preparing
>Thread-1827::INFO::2012-08-30 12:59:39,307::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1827::INFO::2012-08-30 12:59:39,308::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1827::DEBUG::2012-08-30 12:59:39,308::task::1172::TaskManager.Task::(prepare) Task=`61139ffc-49c5-4e66-80f3-a1897d203d26`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1827::DEBUG::2012-08-30 12:59:39,308::task::588::TaskManager.Task::(_updateState) Task=`61139ffc-49c5-4e66-80f3-a1897d203d26`::moving from state preparing -> state finished
>Thread-1827::DEBUG::2012-08-30 12:59:39,308::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1827::DEBUG::2012-08-30 12:59:39,309::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1827::DEBUG::2012-08-30 12:59:39,309::task::978::TaskManager.Task::(_decref) Task=`61139ffc-49c5-4e66-80f3-a1897d203d26`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:59:40,331::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1828::DEBUG::2012-08-30 12:59:42,579::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1828::DEBUG::2012-08-30 12:59:42,580::task::588::TaskManager.Task::(_updateState) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::moving from state init -> state preparing
>Thread-1828::INFO::2012-08-30 12:59:42,580::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1828::DEBUG::2012-08-30 12:59:42,580::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d83eb4a9-2468-437f-85cf-3dfc51876d67`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1828::DEBUG::2012-08-30 12:59:42,581::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1828::DEBUG::2012-08-30 12:59:42,581::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1828::DEBUG::2012-08-30 12:59:42,581::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d83eb4a9-2468-437f-85cf-3dfc51876d67`::Granted request
>Thread-1828::DEBUG::2012-08-30 12:59:42,581::task::817::TaskManager.Task::(resourceAcquired) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1828::DEBUG::2012-08-30 12:59:42,582::task::978::TaskManager.Task::(_decref) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::ref 1 aborting False
>Thread-1828::INFO::2012-08-30 12:59:42,585::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1828::DEBUG::2012-08-30 12:59:42,585::task::1172::TaskManager.Task::(prepare) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1828::DEBUG::2012-08-30 12:59:42,585::task::588::TaskManager.Task::(_updateState) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::moving from state preparing -> state finished
>Thread-1828::DEBUG::2012-08-30 12:59:42,585::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1828::DEBUG::2012-08-30 12:59:42,586::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1828::DEBUG::2012-08-30 12:59:42,587::task::978::TaskManager.Task::(_decref) Task=`cb909945-0402-46ec-bf58-d82c6ca4e405`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:59:43,288::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:45,332::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:45,333::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:48,289::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 12:59:50,334::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 12:59:53,290::task::588::TaskManager.Task::(_updateState) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 12:59:53,291::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 12:59:53,291::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c90d95b1-46e3-4fec-82fd-345a70934052`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 12:59:53,291::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 12:59:53,291::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 12:59:53,292::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c90d95b1-46e3-4fec-82fd-345a70934052`::Granted request
>Thread-40::DEBUG::2012-08-30 12:59:53,292::task::817::TaskManager.Task::(resourceAcquired) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 12:59:53,292::task::978::TaskManager.Task::(_decref) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 12:59:53,294::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 12:59:53,296::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 12:59:53,297::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:59:53,297::task::1172::TaskManager.Task::(prepare) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 12:59:53,298::task::588::TaskManager.Task::(_updateState) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 12:59:53,298::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 12:59:53,298::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 12:59:53,298::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 12:59:53,299::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 12:59:53,299::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 12:59:53,299::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 12:59:53,299::task::978::TaskManager.Task::(_decref) Task=`4f047ca9-69c5-46cb-b8b1-f8a1786f5b0b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 12:59:53,300::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:53,300::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:53,301::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 12:59:53,301::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1832::DEBUG::2012-08-30 12:59:55,288::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1831::DEBUG::2012-08-30 12:59:55,288::task::588::TaskManager.Task::(_updateState) Task=`d6d6894b-478a-4fbe-887c-6ef0d1b596a4`::moving from state init -> state preparing
>Thread-1832::DEBUG::2012-08-30 12:59:55,288::task::588::TaskManager.Task::(_updateState) Task=`c23300cf-587a-409e-9ddf-e701fa98d4e8`::moving from state init -> state preparing
>Thread-1831::INFO::2012-08-30 12:59:55,289::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1832::INFO::2012-08-30 12:59:55,289::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1831::INFO::2012-08-30 12:59:55,289::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0025589466095', 'lastCheck': 1346302790.081907, 'code': 0, 'valid': True}}
>Thread-1832::INFO::2012-08-30 12:59:55,289::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1831::DEBUG::2012-08-30 12:59:55,290::task::1172::TaskManager.Task::(prepare) Task=`d6d6894b-478a-4fbe-887c-6ef0d1b596a4`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0025589466095', 'lastCheck': 1346302790.081907, 'code': 0, 'valid': True}}
>Thread-1832::DEBUG::2012-08-30 12:59:55,290::task::1172::TaskManager.Task::(prepare) Task=`c23300cf-587a-409e-9ddf-e701fa98d4e8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1831::DEBUG::2012-08-30 12:59:55,290::task::588::TaskManager.Task::(_updateState) Task=`d6d6894b-478a-4fbe-887c-6ef0d1b596a4`::moving from state preparing -> state finished
>Thread-1832::DEBUG::2012-08-30 12:59:55,290::task::588::TaskManager.Task::(_updateState) Task=`c23300cf-587a-409e-9ddf-e701fa98d4e8`::moving from state preparing -> state finished
>Thread-1831::DEBUG::2012-08-30 12:59:55,290::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1832::DEBUG::2012-08-30 12:59:55,291::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1831::DEBUG::2012-08-30 12:59:55,291::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1832::DEBUG::2012-08-30 12:59:55,291::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1831::DEBUG::2012-08-30 12:59:55,291::task::978::TaskManager.Task::(_decref) Task=`d6d6894b-478a-4fbe-887c-6ef0d1b596a4`::ref 0 aborting False
>Thread-1832::DEBUG::2012-08-30 12:59:55,291::task::978::TaskManager.Task::(_decref) Task=`c23300cf-587a-409e-9ddf-e701fa98d4e8`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 12:59:55,335::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 12:59:55,512::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 12:59:58,302::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1834::DEBUG::2012-08-30 12:59:58,496::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1834::DEBUG::2012-08-30 12:59:58,496::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1833::DEBUG::2012-08-30 12:59:58,499::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1833::DEBUG::2012-08-30 12:59:58,499::task::588::TaskManager.Task::(_updateState) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::moving from state init -> state preparing
>Thread-1833::INFO::2012-08-30 12:59:58,499::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1833::DEBUG::2012-08-30 12:59:58,500::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f07bb43d-fe60-45f0-ab6d-919a72eff272`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1833::DEBUG::2012-08-30 12:59:58,500::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1833::DEBUG::2012-08-30 12:59:58,500::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1833::DEBUG::2012-08-30 12:59:58,500::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f07bb43d-fe60-45f0-ab6d-919a72eff272`::Granted request
>Thread-1833::DEBUG::2012-08-30 12:59:58,501::task::817::TaskManager.Task::(resourceAcquired) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1833::DEBUG::2012-08-30 12:59:58,501::task::978::TaskManager.Task::(_decref) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::ref 1 aborting False
>Thread-1833::INFO::2012-08-30 12:59:58,504::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1833::DEBUG::2012-08-30 12:59:58,504::task::1172::TaskManager.Task::(prepare) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1833::DEBUG::2012-08-30 12:59:58,504::task::588::TaskManager.Task::(_updateState) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::moving from state preparing -> state finished
>Thread-1833::DEBUG::2012-08-30 12:59:58,505::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1833::DEBUG::2012-08-30 12:59:58,505::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1833::DEBUG::2012-08-30 12:59:58,505::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1833::DEBUG::2012-08-30 12:59:58,505::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1833::DEBUG::2012-08-30 12:59:58,506::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1833::DEBUG::2012-08-30 12:59:58,506::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1833::DEBUG::2012-08-30 12:59:58,506::task::978::TaskManager.Task::(_decref) Task=`c61962e9-c59d-44b2-9539-078d00f8fd82`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:00:00,337::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:00,337::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:00:02,520::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:00:03,303::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:05,338::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:08,304::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:08,304::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:10,339::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1837::DEBUG::2012-08-30 13:00:11,819::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1837::DEBUG::2012-08-30 13:00:11,819::task::588::TaskManager.Task::(_updateState) Task=`dcf03c10-d1ec-4f84-b6cb-cbec6b7735d9`::moving from state init -> state preparing
>Thread-1837::INFO::2012-08-30 13:00:11,819::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1837::INFO::2012-08-30 13:00:11,820::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1837::DEBUG::2012-08-30 13:00:11,820::task::1172::TaskManager.Task::(prepare) Task=`dcf03c10-d1ec-4f84-b6cb-cbec6b7735d9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1837::DEBUG::2012-08-30 13:00:11,820::task::588::TaskManager.Task::(_updateState) Task=`dcf03c10-d1ec-4f84-b6cb-cbec6b7735d9`::moving from state preparing -> state finished
>Thread-1837::DEBUG::2012-08-30 13:00:11,820::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1837::DEBUG::2012-08-30 13:00:11,820::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1837::DEBUG::2012-08-30 13:00:11,821::task::978::TaskManager.Task::(_decref) Task=`dcf03c10-d1ec-4f84-b6cb-cbec6b7735d9`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:00:13,305::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1839::DEBUG::2012-08-30 13:00:15,030::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1839::DEBUG::2012-08-30 13:00:15,030::task::588::TaskManager.Task::(_updateState) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::moving from state init -> state preparing
>Thread-1839::INFO::2012-08-30 13:00:15,031::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1839::DEBUG::2012-08-30 13:00:15,031::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2054d869-d99d-4567-8134-83af18286b29`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1839::DEBUG::2012-08-30 13:00:15,031::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1839::DEBUG::2012-08-30 13:00:15,032::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1839::DEBUG::2012-08-30 13:00:15,032::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2054d869-d99d-4567-8134-83af18286b29`::Granted request
>Thread-1839::DEBUG::2012-08-30 13:00:15,032::task::817::TaskManager.Task::(resourceAcquired) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1839::DEBUG::2012-08-30 13:00:15,032::task::978::TaskManager.Task::(_decref) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::ref 1 aborting False
>Thread-1839::INFO::2012-08-30 13:00:15,035::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1839::DEBUG::2012-08-30 13:00:15,035::task::1172::TaskManager.Task::(prepare) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1839::DEBUG::2012-08-30 13:00:15,036::task::588::TaskManager.Task::(_updateState) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::moving from state preparing -> state finished
>Thread-1839::DEBUG::2012-08-30 13:00:15,036::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1839::DEBUG::2012-08-30 13:00:15,036::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1839::DEBUG::2012-08-30 13:00:15,037::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1839::DEBUG::2012-08-30 13:00:15,037::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1839::DEBUG::2012-08-30 13:00:15,037::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1839::DEBUG::2012-08-30 13:00:15,037::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1839::DEBUG::2012-08-30 13:00:15,037::task::978::TaskManager.Task::(_decref) Task=`42188e4f-2f2f-4bde-9d68-5ff23467c095`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:00:15,340::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:15,340::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:18,307::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:20,341::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:23,308::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:23,309::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1841::DEBUG::2012-08-30 13:00:24,975::task::588::TaskManager.Task::(_updateState) Task=`5d2be228-e3a4-4d30-bfe1-eca937fa1a77`::moving from state init -> state preparing
>Thread-1841::INFO::2012-08-30 13:00:24,975::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1841::INFO::2012-08-30 13:00:24,975::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161981582642', 'lastCheck': 1346302820.101984, 'code': 0, 'valid': True}}
>Thread-1841::DEBUG::2012-08-30 13:00:24,975::task::1172::TaskManager.Task::(prepare) Task=`5d2be228-e3a4-4d30-bfe1-eca937fa1a77`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161981582642', 'lastCheck': 1346302820.101984, 'code': 0, 'valid': True}}
>Thread-1841::DEBUG::2012-08-30 13:00:24,976::task::588::TaskManager.Task::(_updateState) Task=`5d2be228-e3a4-4d30-bfe1-eca937fa1a77`::moving from state preparing -> state finished
>Thread-1841::DEBUG::2012-08-30 13:00:24,976::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1841::DEBUG::2012-08-30 13:00:24,976::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1841::DEBUG::2012-08-30 13:00:24,976::task::978::TaskManager.Task::(_decref) Task=`5d2be228-e3a4-4d30-bfe1-eca937fa1a77`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:00:25,342::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:00:25,540::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:00:28,310::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-26::DEBUG::2012-08-30 13:00:30,102::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:00:30,102::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-26::DEBUG::2012-08-30 13:00:30,103::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None)
>Thread-26::DEBUG::2012-08-30 13:00:30,213::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = '  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5
>Thread-26::WARNING::2012-08-30 13:00:30,214::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found']
>Thread-26::DEBUG::2012-08-30 13:00:30,214::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-26::DEBUG::2012-08-30 13:00:30,220::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:00:30,221::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend
>Thread-26::DEBUG::2012-08-30 13:00:30,223::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d']
>Thread-26::DEBUG::2012-08-30 13:00:30,225::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-26::WARNING::2012-08-30 13:00:30,225::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered
>Thread-26::WARNING::2012-08-30 13:00:30,225::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered
>Thread-1842::DEBUG::2012-08-30 13:00:30,255::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1842::DEBUG::2012-08-30 13:00:30,256::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1843::DEBUG::2012-08-30 13:00:30,256::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1843::DEBUG::2012-08-30 13:00:30,257::task::588::TaskManager.Task::(_updateState) Task=`62bc07d9-3439-4390-a42b-4059a5ae9981`::moving from state init -> state preparing
>Thread-1843::INFO::2012-08-30 13:00:30,258::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1843::INFO::2012-08-30 13:00:30,258::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1843::DEBUG::2012-08-30 13:00:30,259::task::1172::TaskManager.Task::(prepare) Task=`62bc07d9-3439-4390-a42b-4059a5ae9981`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1843::DEBUG::2012-08-30 13:00:30,259::task::588::TaskManager.Task::(_updateState) Task=`62bc07d9-3439-4390-a42b-4059a5ae9981`::moving from state preparing -> state finished
>Thread-1843::DEBUG::2012-08-30 13:00:30,259::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1843::DEBUG::2012-08-30 13:00:30,259::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1843::DEBUG::2012-08-30 13:00:30,260::task::978::TaskManager.Task::(_decref) Task=`62bc07d9-3439-4390-a42b-4059a5ae9981`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 13:00:30,343::task::588::TaskManager.Task::(_updateState) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:00:30,343::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:00:30,344::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a7e14fd6-a963-4831-9a04-ce61465bb232`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:00:30,344::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:00:30,344::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:00:30,345::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a7e14fd6-a963-4831-9a04-ce61465bb232`::Granted request
>Thread-104::DEBUG::2012-08-30 13:00:30,345::task::817::TaskManager.Task::(resourceAcquired) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:00:30,345::task::978::TaskManager.Task::(_decref) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:00:30,347::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:00:30,349::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:00:30,350::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:00:30,350::task::1172::TaskManager.Task::(prepare) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:00:30,350::task::588::TaskManager.Task::(_updateState) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:00:30,351::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:00:30,351::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:00:30,351::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:00:30,351::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:00:30,352::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:00:30,352::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:00:30,352::task::978::TaskManager.Task::(_decref) Task=`7a7b6471-71d3-4e4c-9792-b943c0a65f49`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:00:30,352::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:30,353::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:30,353::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:30,354::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:00:32,546::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:00:33,311::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1849::DEBUG::2012-08-30 13:00:33,668::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1849::DEBUG::2012-08-30 13:00:33,669::task::588::TaskManager.Task::(_updateState) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::moving from state init -> state preparing
>Thread-1849::INFO::2012-08-30 13:00:33,669::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1849::DEBUG::2012-08-30 13:00:33,669::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`aa45b25a-b941-4f85-b7ab-5334500e9cf8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1849::DEBUG::2012-08-30 13:00:33,670::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1849::DEBUG::2012-08-30 13:00:33,670::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1849::DEBUG::2012-08-30 13:00:33,670::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`aa45b25a-b941-4f85-b7ab-5334500e9cf8`::Granted request
>Thread-1849::DEBUG::2012-08-30 13:00:33,670::task::817::TaskManager.Task::(resourceAcquired) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1849::DEBUG::2012-08-30 13:00:33,671::task::978::TaskManager.Task::(_decref) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::ref 1 aborting False
>Thread-1849::INFO::2012-08-30 13:00:33,674::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1849::DEBUG::2012-08-30 13:00:33,674::task::1172::TaskManager.Task::(prepare) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1849::DEBUG::2012-08-30 13:00:33,674::task::588::TaskManager.Task::(_updateState) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::moving from state preparing -> state finished
>Thread-1849::DEBUG::2012-08-30 13:00:33,674::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1849::DEBUG::2012-08-30 13:00:33,675::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1849::DEBUG::2012-08-30 13:00:33,675::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1849::DEBUG::2012-08-30 13:00:33,675::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1849::DEBUG::2012-08-30 13:00:33,675::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1849::DEBUG::2012-08-30 13:00:33,676::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1849::DEBUG::2012-08-30 13:00:33,676::task::978::TaskManager.Task::(_decref) Task=`c5224c28-7d72-42f5-a3ca-05a79208e313`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:00:35,355::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:38,313::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:38,313::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:40,356::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:43,314::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:45,357::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:45,357::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1853::DEBUG::2012-08-30 13:00:46,915::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1853::DEBUG::2012-08-30 13:00:46,915::task::588::TaskManager.Task::(_updateState) Task=`b2e3615e-59a6-4398-a9fc-a2069e22f591`::moving from state init -> state preparing
>Thread-1853::INFO::2012-08-30 13:00:46,915::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1853::INFO::2012-08-30 13:00:46,916::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1853::DEBUG::2012-08-30 13:00:46,916::task::1172::TaskManager.Task::(prepare) Task=`b2e3615e-59a6-4398-a9fc-a2069e22f591`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1853::DEBUG::2012-08-30 13:00:46,916::task::588::TaskManager.Task::(_updateState) Task=`b2e3615e-59a6-4398-a9fc-a2069e22f591`::moving from state preparing -> state finished
>Thread-1853::DEBUG::2012-08-30 13:00:46,916::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1853::DEBUG::2012-08-30 13:00:46,916::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1853::DEBUG::2012-08-30 13:00:46,917::task::978::TaskManager.Task::(_decref) Task=`b2e3615e-59a6-4398-a9fc-a2069e22f591`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:00:48,316::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1854::DEBUG::2012-08-30 13:00:50,117::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1854::DEBUG::2012-08-30 13:00:50,117::task::588::TaskManager.Task::(_updateState) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::moving from state init -> state preparing
>Thread-1854::INFO::2012-08-30 13:00:50,118::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1854::DEBUG::2012-08-30 13:00:50,118::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`56543bce-781a-4421-b8b3-014c80183452`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1854::DEBUG::2012-08-30 13:00:50,118::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1854::DEBUG::2012-08-30 13:00:50,119::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1854::DEBUG::2012-08-30 13:00:50,119::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`56543bce-781a-4421-b8b3-014c80183452`::Granted request
>Thread-1854::DEBUG::2012-08-30 13:00:50,119::task::817::TaskManager.Task::(resourceAcquired) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1854::DEBUG::2012-08-30 13:00:50,119::task::978::TaskManager.Task::(_decref) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::ref 1 aborting False
>Thread-1854::INFO::2012-08-30 13:00:50,122::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1854::DEBUG::2012-08-30 13:00:50,123::task::1172::TaskManager.Task::(prepare) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1854::DEBUG::2012-08-30 13:00:50,123::task::588::TaskManager.Task::(_updateState) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::moving from state preparing -> state finished
>Thread-1854::DEBUG::2012-08-30 13:00:50,123::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1854::DEBUG::2012-08-30 13:00:50,123::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1854::DEBUG::2012-08-30 13:00:50,124::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1854::DEBUG::2012-08-30 13:00:50,124::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1854::DEBUG::2012-08-30 13:00:50,124::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1854::DEBUG::2012-08-30 13:00:50,124::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1854::DEBUG::2012-08-30 13:00:50,125::task::978::TaskManager.Task::(_decref) Task=`c674cd9d-759f-482a-adb7-06c54f1aa3f0`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:00:50,358::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:00:53,316::task::588::TaskManager.Task::(_updateState) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:00:53,317::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:00:53,317::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d223f689-ef3d-4e0e-b121-b5d5f8451a96`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:00:53,318::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:00:53,318::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:00:53,318::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d223f689-ef3d-4e0e-b121-b5d5f8451a96`::Granted request
>Thread-40::DEBUG::2012-08-30 13:00:53,318::task::817::TaskManager.Task::(resourceAcquired) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:00:53,319::task::978::TaskManager.Task::(_decref) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:00:53,320::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:00:53,322::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:00:53,324::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:00:53,324::task::1172::TaskManager.Task::(prepare) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:00:53,324::task::588::TaskManager.Task::(_updateState) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:00:53,324::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:00:53,324::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:00:53,325::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:00:53,325::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:00:53,325::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:00:53,325::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:00:53,326::task::978::TaskManager.Task::(_decref) Task=`84e68be4-bb35-4cc0-a3da-4e2cc9803445`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:00:53,326::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:53,326::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:53,327::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:00:53,327::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:00:55,359::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1856::DEBUG::2012-08-30 13:00:55,368::task::588::TaskManager.Task::(_updateState) Task=`18ad438a-9f67-4869-ab46-6dc621bac2e0`::moving from state init -> state preparing
>Thread-1856::INFO::2012-08-30 13:00:55,368::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1856::INFO::2012-08-30 13:00:55,368::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177001953125', 'lastCheck': 1346302850.244632, 'code': 0, 'valid': True}}
>Thread-1856::DEBUG::2012-08-30 13:00:55,368::task::1172::TaskManager.Task::(prepare) Task=`18ad438a-9f67-4869-ab46-6dc621bac2e0`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177001953125', 'lastCheck': 1346302850.244632, 'code': 0, 'valid': True}}
>Thread-1856::DEBUG::2012-08-30 13:00:55,369::task::588::TaskManager.Task::(_updateState) Task=`18ad438a-9f67-4869-ab46-6dc621bac2e0`::moving from state preparing -> state finished
>Thread-1856::DEBUG::2012-08-30 13:00:55,369::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1856::DEBUG::2012-08-30 13:00:55,369::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1856::DEBUG::2012-08-30 13:00:55,369::task::978::TaskManager.Task::(_decref) Task=`18ad438a-9f67-4869-ab46-6dc621bac2e0`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:00:55,567::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:00:58,328::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:00,360::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:00,360::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1857::DEBUG::2012-08-30 13:01:00,710::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1857::DEBUG::2012-08-30 13:01:00,711::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>VM Channels Listener::DEBUG::2012-08-30 13:01:02,571::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:01:03,330::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1858::DEBUG::2012-08-30 13:01:03,524::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1858::DEBUG::2012-08-30 13:01:03,524::task::588::TaskManager.Task::(_updateState) Task=`68cc0cc0-92e7-40b6-850d-aa4bef7a3a33`::moving from state init -> state preparing
>Thread-1858::INFO::2012-08-30 13:01:03,525::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1858::INFO::2012-08-30 13:01:03,525::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1858::DEBUG::2012-08-30 13:01:03,525::task::1172::TaskManager.Task::(prepare) Task=`68cc0cc0-92e7-40b6-850d-aa4bef7a3a33`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1858::DEBUG::2012-08-30 13:01:03,525::task::588::TaskManager.Task::(_updateState) Task=`68cc0cc0-92e7-40b6-850d-aa4bef7a3a33`::moving from state preparing -> state finished
>Thread-1858::DEBUG::2012-08-30 13:01:03,526::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1858::DEBUG::2012-08-30 13:01:03,526::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1858::DEBUG::2012-08-30 13:01:03,526::task::978::TaskManager.Task::(_decref) Task=`68cc0cc0-92e7-40b6-850d-aa4bef7a3a33`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:01:05,361::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1860::DEBUG::2012-08-30 13:01:06,778::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1860::DEBUG::2012-08-30 13:01:06,779::task::588::TaskManager.Task::(_updateState) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::moving from state init -> state preparing
>Thread-1860::INFO::2012-08-30 13:01:06,779::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1860::DEBUG::2012-08-30 13:01:06,779::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6913cea6-c059-4842-9c9d-b59fea28eae7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1860::DEBUG::2012-08-30 13:01:06,780::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1860::DEBUG::2012-08-30 13:01:06,780::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1860::DEBUG::2012-08-30 13:01:06,780::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6913cea6-c059-4842-9c9d-b59fea28eae7`::Granted request
>Thread-1860::DEBUG::2012-08-30 13:01:06,780::task::817::TaskManager.Task::(resourceAcquired) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1860::DEBUG::2012-08-30 13:01:06,781::task::978::TaskManager.Task::(_decref) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::ref 1 aborting False
>Thread-1860::INFO::2012-08-30 13:01:06,784::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1860::DEBUG::2012-08-30 13:01:06,784::task::1172::TaskManager.Task::(prepare) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1860::DEBUG::2012-08-30 13:01:06,784::task::588::TaskManager.Task::(_updateState) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::moving from state preparing -> state finished
>Thread-1860::DEBUG::2012-08-30 13:01:06,784::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1860::DEBUG::2012-08-30 13:01:06,784::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1860::DEBUG::2012-08-30 13:01:06,785::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1860::DEBUG::2012-08-30 13:01:06,785::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1860::DEBUG::2012-08-30 13:01:06,785::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1860::DEBUG::2012-08-30 13:01:06,785::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1860::DEBUG::2012-08-30 13:01:06,786::task::978::TaskManager.Task::(_decref) Task=`7d9c99a0-ae86-41fc-826c-7a6624fb3218`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:01:08,331::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:08,332::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:10,362::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:13,333::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:15,363::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:15,364::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:18,334::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1863::DEBUG::2012-08-30 13:01:20,031::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1863::DEBUG::2012-08-30 13:01:20,032::task::588::TaskManager.Task::(_updateState) Task=`b5460281-83ee-44c3-aff4-af80d9e4f6b9`::moving from state init -> state preparing
>Thread-1863::INFO::2012-08-30 13:01:20,032::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1863::INFO::2012-08-30 13:01:20,032::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1863::DEBUG::2012-08-30 13:01:20,032::task::1172::TaskManager.Task::(prepare) Task=`b5460281-83ee-44c3-aff4-af80d9e4f6b9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1863::DEBUG::2012-08-30 13:01:20,033::task::588::TaskManager.Task::(_updateState) Task=`b5460281-83ee-44c3-aff4-af80d9e4f6b9`::moving from state preparing -> state finished
>Thread-1863::DEBUG::2012-08-30 13:01:20,033::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1863::DEBUG::2012-08-30 13:01:20,033::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1863::DEBUG::2012-08-30 13:01:20,033::task::978::TaskManager.Task::(_decref) Task=`b5460281-83ee-44c3-aff4-af80d9e4f6b9`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:01:20,365::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:23,336::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:23,336::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1865::DEBUG::2012-08-30 13:01:25,129::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1865::DEBUG::2012-08-30 13:01:25,129::task::588::TaskManager.Task::(_updateState) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::moving from state init -> state preparing
>Thread-1865::INFO::2012-08-30 13:01:25,129::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1865::DEBUG::2012-08-30 13:01:25,130::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`dd01ec3d-337f-4269-b847-7784c7747b8f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1865::DEBUG::2012-08-30 13:01:25,130::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1865::DEBUG::2012-08-30 13:01:25,130::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1865::DEBUG::2012-08-30 13:01:25,130::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`dd01ec3d-337f-4269-b847-7784c7747b8f`::Granted request
>Thread-1865::DEBUG::2012-08-30 13:01:25,131::task::817::TaskManager.Task::(resourceAcquired) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1865::DEBUG::2012-08-30 13:01:25,131::task::978::TaskManager.Task::(_decref) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::ref 1 aborting False
>Thread-1865::INFO::2012-08-30 13:01:25,134::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1865::DEBUG::2012-08-30 13:01:25,134::task::1172::TaskManager.Task::(prepare) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1865::DEBUG::2012-08-30 13:01:25,134::task::588::TaskManager.Task::(_updateState) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::moving from state preparing -> state finished
>Thread-1865::DEBUG::2012-08-30 13:01:25,135::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1865::DEBUG::2012-08-30 13:01:25,135::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1865::DEBUG::2012-08-30 13:01:25,135::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1865::DEBUG::2012-08-30 13:01:25,135::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1865::DEBUG::2012-08-30 13:01:25,136::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1865::DEBUG::2012-08-30 13:01:25,136::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1865::DEBUG::2012-08-30 13:01:25,136::task::978::TaskManager.Task::(_decref) Task=`1c6ab7af-bd5c-4b2d-baa9-35b3d9006a98`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:01:25,365::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:01:25,590::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1866::DEBUG::2012-08-30 13:01:26,768::task::588::TaskManager.Task::(_updateState) Task=`b4ef5710-9499-46bc-9c16-fb382445a835`::moving from state init -> state preparing
>Thread-1866::INFO::2012-08-30 13:01:26,768::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1866::INFO::2012-08-30 13:01:26,768::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162816047668', 'lastCheck': 1346302880.2646329, 'code': 0, 'valid': True}}
>Thread-1866::DEBUG::2012-08-30 13:01:26,768::task::1172::TaskManager.Task::(prepare) Task=`b4ef5710-9499-46bc-9c16-fb382445a835`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162816047668', 'lastCheck': 1346302880.2646329, 'code': 0, 'valid': True}}
>Thread-1866::DEBUG::2012-08-30 13:01:26,769::task::588::TaskManager.Task::(_updateState) Task=`b4ef5710-9499-46bc-9c16-fb382445a835`::moving from state preparing -> state finished
>Thread-1866::DEBUG::2012-08-30 13:01:26,769::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1866::DEBUG::2012-08-30 13:01:26,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1866::DEBUG::2012-08-30 13:01:26,769::task::978::TaskManager.Task::(_decref) Task=`b4ef5710-9499-46bc-9c16-fb382445a835`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:01:28,337::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1867::DEBUG::2012-08-30 13:01:30,082::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1867::DEBUG::2012-08-30 13:01:30,082::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::DEBUG::2012-08-30 13:01:30,366::task::588::TaskManager.Task::(_updateState) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:01:30,367::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:01:30,367::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`eef7b159-3f07-42b7-905a-0937a3379e5b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:01:30,367::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:01:30,368::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:01:30,368::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`eef7b159-3f07-42b7-905a-0937a3379e5b`::Granted request
>Thread-104::DEBUG::2012-08-30 13:01:30,368::task::817::TaskManager.Task::(resourceAcquired) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:01:30,368::task::978::TaskManager.Task::(_decref) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:01:30,370::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:01:30,372::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:01:30,373::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:01:30,373::task::1172::TaskManager.Task::(prepare) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:01:30,373::task::588::TaskManager.Task::(_updateState) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:01:30,374::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:01:30,374::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:01:30,374::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:01:30,374::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:01:30,375::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:01:30,375::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:01:30,375::task::978::TaskManager.Task::(_decref) Task=`b9558f7b-2e65-4643-b432-654d1e72017b`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:01:30,375::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:30,376::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:30,376::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:30,377::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:01:32,593::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:01:33,339::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:35,378::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:38,340::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:38,340::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1869::DEBUG::2012-08-30 13:01:38,427::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1869::DEBUG::2012-08-30 13:01:38,427::task::588::TaskManager.Task::(_updateState) Task=`9705d26c-d7bb-4095-975c-4c356c4c063e`::moving from state init -> state preparing
>Thread-1869::INFO::2012-08-30 13:01:38,428::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1869::INFO::2012-08-30 13:01:38,428::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1869::DEBUG::2012-08-30 13:01:38,428::task::1172::TaskManager.Task::(prepare) Task=`9705d26c-d7bb-4095-975c-4c356c4c063e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1869::DEBUG::2012-08-30 13:01:38,428::task::588::TaskManager.Task::(_updateState) Task=`9705d26c-d7bb-4095-975c-4c356c4c063e`::moving from state preparing -> state finished
>Thread-1869::DEBUG::2012-08-30 13:01:38,428::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1869::DEBUG::2012-08-30 13:01:38,429::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1869::DEBUG::2012-08-30 13:01:38,429::task::978::TaskManager.Task::(_decref) Task=`9705d26c-d7bb-4095-975c-4c356c4c063e`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:01:40,379::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1871::DEBUG::2012-08-30 13:01:41,795::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1871::DEBUG::2012-08-30 13:01:41,795::task::588::TaskManager.Task::(_updateState) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::moving from state init -> state preparing
>Thread-1871::INFO::2012-08-30 13:01:41,796::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1871::DEBUG::2012-08-30 13:01:41,796::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`782091b4-c9ab-4175-a65b-1a66ba270615`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1871::DEBUG::2012-08-30 13:01:41,796::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1871::DEBUG::2012-08-30 13:01:41,796::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1871::DEBUG::2012-08-30 13:01:41,797::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`782091b4-c9ab-4175-a65b-1a66ba270615`::Granted request
>Thread-1871::DEBUG::2012-08-30 13:01:41,797::task::817::TaskManager.Task::(resourceAcquired) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1871::DEBUG::2012-08-30 13:01:41,797::task::978::TaskManager.Task::(_decref) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::ref 1 aborting False
>Thread-1871::INFO::2012-08-30 13:01:41,800::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1871::DEBUG::2012-08-30 13:01:41,800::task::1172::TaskManager.Task::(prepare) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1871::DEBUG::2012-08-30 13:01:41,801::task::588::TaskManager.Task::(_updateState) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::moving from state preparing -> state finished
>Thread-1871::DEBUG::2012-08-30 13:01:41,801::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1871::DEBUG::2012-08-30 13:01:41,801::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1871::DEBUG::2012-08-30 13:01:41,801::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1871::DEBUG::2012-08-30 13:01:41,802::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1871::DEBUG::2012-08-30 13:01:41,802::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1871::DEBUG::2012-08-30 13:01:41,802::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1871::DEBUG::2012-08-30 13:01:41,802::task::978::TaskManager.Task::(_decref) Task=`c6c5dcb0-b110-4c77-b6ab-3a5d20e5082c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:01:43,341::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:45,380::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:45,380::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:48,343::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:01:50,381::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:01:53,344::task::588::TaskManager.Task::(_updateState) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:01:53,344::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:01:53,345::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c61d42a6-ba38-453a-be83-06a22ca69600`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:01:53,345::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:01:53,345::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:01:53,345::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c61d42a6-ba38-453a-be83-06a22ca69600`::Granted request
>Thread-40::DEBUG::2012-08-30 13:01:53,346::task::817::TaskManager.Task::(resourceAcquired) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:01:53,346::task::978::TaskManager.Task::(_decref) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:01:53,348::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:01:53,350::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:01:53,351::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:01:53,351::task::1172::TaskManager.Task::(prepare) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:01:53,351::task::588::TaskManager.Task::(_updateState) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:01:53,352::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:01:53,352::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:01:53,352::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:01:53,352::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:01:53,353::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:01:53,353::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:01:53,353::task::978::TaskManager.Task::(_decref) Task=`35cb29b5-332c-4160-ae29-522e56c406e8`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:01:53,354::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:53,354::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:53,354::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:01:53,355::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1874::DEBUG::2012-08-30 13:01:55,125::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1874::DEBUG::2012-08-30 13:01:55,125::task::588::TaskManager.Task::(_updateState) Task=`e3ddc274-97ff-48a6-8808-cee8f2a115b8`::moving from state init -> state preparing
>Thread-1874::INFO::2012-08-30 13:01:55,125::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1874::INFO::2012-08-30 13:01:55,126::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1874::DEBUG::2012-08-30 13:01:55,126::task::1172::TaskManager.Task::(prepare) Task=`e3ddc274-97ff-48a6-8808-cee8f2a115b8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1874::DEBUG::2012-08-30 13:01:55,126::task::588::TaskManager.Task::(_updateState) Task=`e3ddc274-97ff-48a6-8808-cee8f2a115b8`::moving from state preparing -> state finished
>Thread-1874::DEBUG::2012-08-30 13:01:55,126::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1874::DEBUG::2012-08-30 13:01:55,126::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1874::DEBUG::2012-08-30 13:01:55,127::task::978::TaskManager.Task::(_decref) Task=`e3ddc274-97ff-48a6-8808-cee8f2a115b8`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:01:55,382::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:01:55,615::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1875::DEBUG::2012-08-30 13:01:55,932::task::588::TaskManager.Task::(_updateState) Task=`884daea6-e683-422a-bf59-dea93fe65d0b`::moving from state init -> state preparing
>Thread-1875::INFO::2012-08-30 13:01:55,932::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1875::INFO::2012-08-30 13:01:55,933::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176906585693', 'lastCheck': 1346302910.284256, 'code': 0, 'valid': True}}
>Thread-1875::DEBUG::2012-08-30 13:01:55,933::task::1172::TaskManager.Task::(prepare) Task=`884daea6-e683-422a-bf59-dea93fe65d0b`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176906585693', 'lastCheck': 1346302910.284256, 'code': 0, 'valid': True}}
>Thread-1875::DEBUG::2012-08-30 13:01:55,933::task::588::TaskManager.Task::(_updateState) Task=`884daea6-e683-422a-bf59-dea93fe65d0b`::moving from state preparing -> state finished
>Thread-1875::DEBUG::2012-08-30 13:01:55,933::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1875::DEBUG::2012-08-30 13:01:55,934::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1875::DEBUG::2012-08-30 13:01:55,934::task::978::TaskManager.Task::(_decref) Task=`884daea6-e683-422a-bf59-dea93fe65d0b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:01:58,356::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1876::DEBUG::2012-08-30 13:01:58,373::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1876::DEBUG::2012-08-30 13:01:58,373::task::588::TaskManager.Task::(_updateState) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::moving from state init -> state preparing
>Thread-1876::INFO::2012-08-30 13:01:58,374::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1876::DEBUG::2012-08-30 13:01:58,374::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e97fa825-53b2-4687-92ea-1130ec2325f7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1876::DEBUG::2012-08-30 13:01:58,374::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1876::DEBUG::2012-08-30 13:01:58,374::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1876::DEBUG::2012-08-30 13:01:58,375::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e97fa825-53b2-4687-92ea-1130ec2325f7`::Granted request
>Thread-1876::DEBUG::2012-08-30 13:01:58,375::task::817::TaskManager.Task::(resourceAcquired) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1876::DEBUG::2012-08-30 13:01:58,375::task::978::TaskManager.Task::(_decref) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::ref 1 aborting False
>Thread-1876::INFO::2012-08-30 13:01:58,378::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1876::DEBUG::2012-08-30 13:01:58,378::task::1172::TaskManager.Task::(prepare) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1876::DEBUG::2012-08-30 13:01:58,379::task::588::TaskManager.Task::(_updateState) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::moving from state preparing -> state finished
>Thread-1876::DEBUG::2012-08-30 13:01:58,379::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1876::DEBUG::2012-08-30 13:01:58,379::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1876::DEBUG::2012-08-30 13:01:58,379::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1876::DEBUG::2012-08-30 13:01:58,380::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1876::DEBUG::2012-08-30 13:01:58,380::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1876::DEBUG::2012-08-30 13:01:58,380::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1876::DEBUG::2012-08-30 13:01:58,380::task::978::TaskManager.Task::(_decref) Task=`95dd036c-bd7c-45a6-aeff-453d88fb9cda`::ref 0 aborting False
>Thread-1877::DEBUG::2012-08-30 13:01:59,216::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1877::DEBUG::2012-08-30 13:01:59,216::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:02:00,383::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:00,383::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:02:02,622::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:02:03,357::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:05,384::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:08,358::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:08,359::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:10,385::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1880::DEBUG::2012-08-30 13:02:11,309::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1880::DEBUG::2012-08-30 13:02:11,309::task::588::TaskManager.Task::(_updateState) Task=`20d392de-1b87-48cb-8299-08a2bc17f94a`::moving from state init -> state preparing
>Thread-1880::INFO::2012-08-30 13:02:11,309::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1880::INFO::2012-08-30 13:02:11,310::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1880::DEBUG::2012-08-30 13:02:11,310::task::1172::TaskManager.Task::(prepare) Task=`20d392de-1b87-48cb-8299-08a2bc17f94a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1880::DEBUG::2012-08-30 13:02:11,310::task::588::TaskManager.Task::(_updateState) Task=`20d392de-1b87-48cb-8299-08a2bc17f94a`::moving from state preparing -> state finished
>Thread-1880::DEBUG::2012-08-30 13:02:11,310::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1880::DEBUG::2012-08-30 13:02:11,310::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1880::DEBUG::2012-08-30 13:02:11,311::task::978::TaskManager.Task::(_decref) Task=`20d392de-1b87-48cb-8299-08a2bc17f94a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:02:13,360::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1881::DEBUG::2012-08-30 13:02:14,568::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1881::DEBUG::2012-08-30 13:02:14,569::task::588::TaskManager.Task::(_updateState) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::moving from state init -> state preparing
>Thread-1881::INFO::2012-08-30 13:02:14,569::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1881::DEBUG::2012-08-30 13:02:14,570::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0f180446-afa3-4a8b-900f-89b9fb1cc0e2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1881::DEBUG::2012-08-30 13:02:14,570::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1881::DEBUG::2012-08-30 13:02:14,570::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1881::DEBUG::2012-08-30 13:02:14,571::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0f180446-afa3-4a8b-900f-89b9fb1cc0e2`::Granted request
>Thread-1881::DEBUG::2012-08-30 13:02:14,571::task::817::TaskManager.Task::(resourceAcquired) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1881::DEBUG::2012-08-30 13:02:14,571::task::978::TaskManager.Task::(_decref) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::ref 1 aborting False
>Thread-1881::INFO::2012-08-30 13:02:14,574::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1881::DEBUG::2012-08-30 13:02:14,574::task::1172::TaskManager.Task::(prepare) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1881::DEBUG::2012-08-30 13:02:14,575::task::588::TaskManager.Task::(_updateState) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::moving from state preparing -> state finished
>Thread-1881::DEBUG::2012-08-30 13:02:14,575::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1881::DEBUG::2012-08-30 13:02:14,575::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1881::DEBUG::2012-08-30 13:02:14,575::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1881::DEBUG::2012-08-30 13:02:14,576::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1881::DEBUG::2012-08-30 13:02:14,576::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1881::DEBUG::2012-08-30 13:02:14,576::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1881::DEBUG::2012-08-30 13:02:14,576::task::978::TaskManager.Task::(_decref) Task=`389b0420-d77f-4f78-8427-ab4e4b633868`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:02:15,386::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:15,387::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:18,361::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:20,388::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:23,363::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:23,363::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1884::DEBUG::2012-08-30 13:02:24,937::task::588::TaskManager.Task::(_updateState) Task=`e1e80c4a-3f45-4352-8c01-445f27f01ebf`::moving from state init -> state preparing
>Thread-1884::INFO::2012-08-30 13:02:24,938::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1884::INFO::2012-08-30 13:02:24,938::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163197517395', 'lastCheck': 1346302940.304249, 'code': 0, 'valid': True}}
>Thread-1884::DEBUG::2012-08-30 13:02:24,938::task::1172::TaskManager.Task::(prepare) Task=`e1e80c4a-3f45-4352-8c01-445f27f01ebf`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163197517395', 'lastCheck': 1346302940.304249, 'code': 0, 'valid': True}}
>Thread-1884::DEBUG::2012-08-30 13:02:24,938::task::588::TaskManager.Task::(_updateState) Task=`e1e80c4a-3f45-4352-8c01-445f27f01ebf`::moving from state preparing -> state finished
>Thread-1884::DEBUG::2012-08-30 13:02:24,939::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1884::DEBUG::2012-08-30 13:02:24,939::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1884::DEBUG::2012-08-30 13:02:24,939::task::978::TaskManager.Task::(_decref) Task=`e1e80c4a-3f45-4352-8c01-445f27f01ebf`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:02:25,389::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:02:25,640::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:02:28,364::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1885::DEBUG::2012-08-30 13:02:30,213::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1885::DEBUG::2012-08-30 13:02:30,213::task::588::TaskManager.Task::(_updateState) Task=`65a724e0-4406-4e8b-ad55-cc3f59e2cd6c`::moving from state init -> state preparing
>Thread-1885::INFO::2012-08-30 13:02:30,213::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1885::INFO::2012-08-30 13:02:30,214::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1885::DEBUG::2012-08-30 13:02:30,214::task::1172::TaskManager.Task::(prepare) Task=`65a724e0-4406-4e8b-ad55-cc3f59e2cd6c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1885::DEBUG::2012-08-30 13:02:30,214::task::588::TaskManager.Task::(_updateState) Task=`65a724e0-4406-4e8b-ad55-cc3f59e2cd6c`::moving from state preparing -> state finished
>Thread-1885::DEBUG::2012-08-30 13:02:30,214::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1885::DEBUG::2012-08-30 13:02:30,215::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1885::DEBUG::2012-08-30 13:02:30,215::task::978::TaskManager.Task::(_decref) Task=`65a724e0-4406-4e8b-ad55-cc3f59e2cd6c`::ref 0 aborting False
>Thread-1886::DEBUG::2012-08-30 13:02:30,216::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1886::DEBUG::2012-08-30 13:02:30,216::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::DEBUG::2012-08-30 13:02:30,390::task::588::TaskManager.Task::(_updateState) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:02:30,390::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:02:30,390::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`43d5a160-00d7-41c2-89e3-3a25256f1b66`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:02:30,390::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:02:30,391::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:02:30,391::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`43d5a160-00d7-41c2-89e3-3a25256f1b66`::Granted request
>Thread-104::DEBUG::2012-08-30 13:02:30,391::task::817::TaskManager.Task::(resourceAcquired) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:02:30,392::task::978::TaskManager.Task::(_decref) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:02:30,393::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:02:30,395::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:02:30,396::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:02:30,396::task::1172::TaskManager.Task::(prepare) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:02:30,397::task::588::TaskManager.Task::(_updateState) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:02:30,397::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:02:30,397::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:02:30,397::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:02:30,397::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:02:30,398::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:02:30,398::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:02:30,398::task::978::TaskManager.Task::(_decref) Task=`769d7947-0a1a-44a0-bab3-d6885c8fe330`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:02:30,399::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:30,399::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:30,399::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:30,400::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:02:32,647::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:02:33,365::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1887::DEBUG::2012-08-30 13:02:33,404::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1887::DEBUG::2012-08-30 13:02:33,404::task::588::TaskManager.Task::(_updateState) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::moving from state init -> state preparing
>Thread-1887::INFO::2012-08-30 13:02:33,404::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1887::DEBUG::2012-08-30 13:02:33,405::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`7ded09c5-e9d2-4c19-9f43-fc63494f4c91`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1887::DEBUG::2012-08-30 13:02:33,405::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1887::DEBUG::2012-08-30 13:02:33,405::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1887::DEBUG::2012-08-30 13:02:33,405::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`7ded09c5-e9d2-4c19-9f43-fc63494f4c91`::Granted request
>Thread-1887::DEBUG::2012-08-30 13:02:33,406::task::817::TaskManager.Task::(resourceAcquired) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1887::DEBUG::2012-08-30 13:02:33,406::task::978::TaskManager.Task::(_decref) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::ref 1 aborting False
>Thread-1887::INFO::2012-08-30 13:02:33,409::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1887::DEBUG::2012-08-30 13:02:33,409::task::1172::TaskManager.Task::(prepare) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1887::DEBUG::2012-08-30 13:02:33,409::task::588::TaskManager.Task::(_updateState) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::moving from state preparing -> state finished
>Thread-1887::DEBUG::2012-08-30 13:02:33,410::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1887::DEBUG::2012-08-30 13:02:33,410::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1887::DEBUG::2012-08-30 13:02:33,410::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1887::DEBUG::2012-08-30 13:02:33,410::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1887::DEBUG::2012-08-30 13:02:33,411::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1887::DEBUG::2012-08-30 13:02:33,411::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1887::DEBUG::2012-08-30 13:02:33,411::task::978::TaskManager.Task::(_decref) Task=`9b5585f8-5f0b-420d-8efb-597dcd8ca558`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:02:35,401::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:38,366::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:38,367::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:40,402::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:43,368::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:45,403::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:45,403::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1891::DEBUG::2012-08-30 13:02:46,855::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1891::DEBUG::2012-08-30 13:02:46,856::task::588::TaskManager.Task::(_updateState) Task=`c0a52f9b-1634-438d-a3e2-5d970bb60bc8`::moving from state init -> state preparing
>Thread-1891::INFO::2012-08-30 13:02:46,856::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1891::INFO::2012-08-30 13:02:46,856::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1891::DEBUG::2012-08-30 13:02:46,857::task::1172::TaskManager.Task::(prepare) Task=`c0a52f9b-1634-438d-a3e2-5d970bb60bc8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1891::DEBUG::2012-08-30 13:02:46,857::task::588::TaskManager.Task::(_updateState) Task=`c0a52f9b-1634-438d-a3e2-5d970bb60bc8`::moving from state preparing -> state finished
>Thread-1891::DEBUG::2012-08-30 13:02:46,857::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1891::DEBUG::2012-08-30 13:02:46,857::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1891::DEBUG::2012-08-30 13:02:46,858::task::978::TaskManager.Task::(_decref) Task=`c0a52f9b-1634-438d-a3e2-5d970bb60bc8`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:02:48,369::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1892::DEBUG::2012-08-30 13:02:50,126::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1892::DEBUG::2012-08-30 13:02:50,126::task::588::TaskManager.Task::(_updateState) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::moving from state init -> state preparing
>Thread-1892::INFO::2012-08-30 13:02:50,127::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1892::DEBUG::2012-08-30 13:02:50,128::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0a77e28f-efe7-4443-b664-02e4dcf76393`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1892::DEBUG::2012-08-30 13:02:50,128::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1892::DEBUG::2012-08-30 13:02:50,128::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1892::DEBUG::2012-08-30 13:02:50,128::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0a77e28f-efe7-4443-b664-02e4dcf76393`::Granted request
>Thread-1892::DEBUG::2012-08-30 13:02:50,129::task::817::TaskManager.Task::(resourceAcquired) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1892::DEBUG::2012-08-30 13:02:50,129::task::978::TaskManager.Task::(_decref) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::ref 1 aborting False
>Thread-1892::INFO::2012-08-30 13:02:50,132::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1892::DEBUG::2012-08-30 13:02:50,132::task::1172::TaskManager.Task::(prepare) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1892::DEBUG::2012-08-30 13:02:50,132::task::588::TaskManager.Task::(_updateState) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::moving from state preparing -> state finished
>Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1892::DEBUG::2012-08-30 13:02:50,133::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1892::DEBUG::2012-08-30 13:02:50,134::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1892::DEBUG::2012-08-30 13:02:50,134::task::978::TaskManager.Task::(_decref) Task=`f4a026bf-477c-4527-93a6-d0db64f24cc2`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:02:50,404::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:02:53,370::task::588::TaskManager.Task::(_updateState) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:02:53,371::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:02:53,371::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`25fd5e2c-d7ec-40ce-bd34-4426fa57d707`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:02:53,372::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:02:53,372::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:02:53,372::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`25fd5e2c-d7ec-40ce-bd34-4426fa57d707`::Granted request
>Thread-40::DEBUG::2012-08-30 13:02:53,372::task::817::TaskManager.Task::(resourceAcquired) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:02:53,373::task::978::TaskManager.Task::(_decref) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:02:53,374::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:02:53,377::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:02:53,378::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:02:53,378::task::1172::TaskManager.Task::(prepare) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:02:53,378::task::588::TaskManager.Task::(_updateState) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:02:53,378::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:02:53,379::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:02:53,380::task::978::TaskManager.Task::(_decref) Task=`63a9b13e-37fc-44e8-9cc4-a08796ac58c9`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:02:53,380::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:53,381::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:53,381::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:02:53,381::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:02:55,405::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1894::DEBUG::2012-08-30 13:02:55,475::task::588::TaskManager.Task::(_updateState) Task=`a716f75a-0b35-41a3-a415-6e7855d10b18`::moving from state init -> state preparing
>Thread-1894::INFO::2012-08-30 13:02:55,475::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1894::INFO::2012-08-30 13:02:55,475::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176095962524', 'lastCheck': 1346302970.323657, 'code': 0, 'valid': True}}
>Thread-1894::DEBUG::2012-08-30 13:02:55,475::task::1172::TaskManager.Task::(prepare) Task=`a716f75a-0b35-41a3-a415-6e7855d10b18`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176095962524', 'lastCheck': 1346302970.323657, 'code': 0, 'valid': True}}
>Thread-1894::DEBUG::2012-08-30 13:02:55,476::task::588::TaskManager.Task::(_updateState) Task=`a716f75a-0b35-41a3-a415-6e7855d10b18`::moving from state preparing -> state finished
>Thread-1894::DEBUG::2012-08-30 13:02:55,476::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1894::DEBUG::2012-08-30 13:02:55,476::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1894::DEBUG::2012-08-30 13:02:55,476::task::978::TaskManager.Task::(_decref) Task=`a716f75a-0b35-41a3-a415-6e7855d10b18`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:02:55,669::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:02:58,382::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1895::DEBUG::2012-08-30 13:02:58,648::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1895::DEBUG::2012-08-30 13:02:58,648::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:03:00,406::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:00,406::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:03:02,676::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-1896::DEBUG::2012-08-30 13:03:03,323::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1896::DEBUG::2012-08-30 13:03:03,324::task::588::TaskManager.Task::(_updateState) Task=`a05ad071-a642-437c-a226-21e9b3d91b55`::moving from state init -> state preparing
>Thread-1896::INFO::2012-08-30 13:03:03,324::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1896::INFO::2012-08-30 13:03:03,325::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1896::DEBUG::2012-08-30 13:03:03,325::task::1172::TaskManager.Task::(prepare) Task=`a05ad071-a642-437c-a226-21e9b3d91b55`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1896::DEBUG::2012-08-30 13:03:03,325::task::588::TaskManager.Task::(_updateState) Task=`a05ad071-a642-437c-a226-21e9b3d91b55`::moving from state preparing -> state finished
>Thread-1896::DEBUG::2012-08-30 13:03:03,325::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1896::DEBUG::2012-08-30 13:03:03,325::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1896::DEBUG::2012-08-30 13:03:03,326::task::978::TaskManager.Task::(_decref) Task=`a05ad071-a642-437c-a226-21e9b3d91b55`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:03:03,383::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:05,407::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1898::DEBUG::2012-08-30 13:03:06,536::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1898::DEBUG::2012-08-30 13:03:06,537::task::588::TaskManager.Task::(_updateState) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::moving from state init -> state preparing
>Thread-1898::INFO::2012-08-30 13:03:06,537::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1898::DEBUG::2012-08-30 13:03:06,537::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2dc91137-d70a-409c-a94b-ce10f0857207`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1898::DEBUG::2012-08-30 13:03:06,538::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1898::DEBUG::2012-08-30 13:03:06,538::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1898::DEBUG::2012-08-30 13:03:06,538::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2dc91137-d70a-409c-a94b-ce10f0857207`::Granted request
>Thread-1898::DEBUG::2012-08-30 13:03:06,538::task::817::TaskManager.Task::(resourceAcquired) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1898::DEBUG::2012-08-30 13:03:06,539::task::978::TaskManager.Task::(_decref) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::ref 1 aborting False
>Thread-1898::INFO::2012-08-30 13:03:06,542::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1898::DEBUG::2012-08-30 13:03:06,542::task::1172::TaskManager.Task::(prepare) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1898::DEBUG::2012-08-30 13:03:06,542::task::588::TaskManager.Task::(_updateState) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::moving from state preparing -> state finished
>Thread-1898::DEBUG::2012-08-30 13:03:06,542::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1898::DEBUG::2012-08-30 13:03:06,543::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1898::DEBUG::2012-08-30 13:03:06,544::task::978::TaskManager.Task::(_decref) Task=`d541fb78-2120-4050-b59c-963a2b2d1977`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:03:08,385::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:08,385::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:10,408::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:13,386::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:15,409::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:15,410::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:18,387::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:20,410::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1902::DEBUG::2012-08-30 13:03:21,817::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1902::DEBUG::2012-08-30 13:03:21,817::task::588::TaskManager.Task::(_updateState) Task=`c83d8300-7370-4725-8681-e580af4687a6`::moving from state init -> state preparing
>Thread-1902::INFO::2012-08-30 13:03:21,818::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1902::INFO::2012-08-30 13:03:21,818::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1902::DEBUG::2012-08-30 13:03:21,818::task::1172::TaskManager.Task::(prepare) Task=`c83d8300-7370-4725-8681-e580af4687a6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1902::DEBUG::2012-08-30 13:03:21,818::task::588::TaskManager.Task::(_updateState) Task=`c83d8300-7370-4725-8681-e580af4687a6`::moving from state preparing -> state finished
>Thread-1902::DEBUG::2012-08-30 13:03:21,819::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1902::DEBUG::2012-08-30 13:03:21,819::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1902::DEBUG::2012-08-30 13:03:21,819::task::978::TaskManager.Task::(_decref) Task=`c83d8300-7370-4725-8681-e580af4687a6`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:03:23,389::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:23,390::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1903::DEBUG::2012-08-30 13:03:24,996::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1903::DEBUG::2012-08-30 13:03:24,996::task::588::TaskManager.Task::(_updateState) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::moving from state init -> state preparing
>Thread-1903::INFO::2012-08-30 13:03:24,996::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1903::DEBUG::2012-08-30 13:03:24,997::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`47b2d6b4-0eb6-415a-ba0c-68a53635ecbc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1903::DEBUG::2012-08-30 13:03:24,997::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1903::DEBUG::2012-08-30 13:03:24,997::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1903::DEBUG::2012-08-30 13:03:24,997::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`47b2d6b4-0eb6-415a-ba0c-68a53635ecbc`::Granted request
>Thread-1903::DEBUG::2012-08-30 13:03:24,998::task::817::TaskManager.Task::(resourceAcquired) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1903::DEBUG::2012-08-30 13:03:24,998::task::978::TaskManager.Task::(_decref) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::ref 1 aborting False
>Thread-1903::INFO::2012-08-30 13:03:25,001::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1903::DEBUG::2012-08-30 13:03:25,001::task::1172::TaskManager.Task::(prepare) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1903::DEBUG::2012-08-30 13:03:25,001::task::588::TaskManager.Task::(_updateState) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::moving from state preparing -> state finished
>Thread-1903::DEBUG::2012-08-30 13:03:25,002::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1903::DEBUG::2012-08-30 13:03:25,002::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1903::DEBUG::2012-08-30 13:03:25,002::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1903::DEBUG::2012-08-30 13:03:25,002::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1903::DEBUG::2012-08-30 13:03:25,003::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1903::DEBUG::2012-08-30 13:03:25,003::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1903::DEBUG::2012-08-30 13:03:25,003::task::978::TaskManager.Task::(_decref) Task=`17c7613b-8bc3-475c-9e55-9cb6a3ab43db`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:03:25,411::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:03:25,697::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1904::DEBUG::2012-08-30 13:03:25,737::task::588::TaskManager.Task::(_updateState) Task=`a1142226-e541-4680-b82a-db4a8c56e568`::moving from state init -> state preparing
>Thread-1904::INFO::2012-08-30 13:03:25,737::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1904::INFO::2012-08-30 13:03:25,738::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162386894226', 'lastCheck': 1346303000.3436971, 'code': 0, 'valid': True}}
>Thread-1904::DEBUG::2012-08-30 13:03:25,738::task::1172::TaskManager.Task::(prepare) Task=`a1142226-e541-4680-b82a-db4a8c56e568`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162386894226', 'lastCheck': 1346303000.3436971, 'code': 0, 'valid': True}}
>Thread-1904::DEBUG::2012-08-30 13:03:25,738::task::588::TaskManager.Task::(_updateState) Task=`a1142226-e541-4680-b82a-db4a8c56e568`::moving from state preparing -> state finished
>Thread-1904::DEBUG::2012-08-30 13:03:25,738::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1904::DEBUG::2012-08-30 13:03:25,738::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1904::DEBUG::2012-08-30 13:03:25,739::task::978::TaskManager.Task::(_decref) Task=`a1142226-e541-4680-b82a-db4a8c56e568`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:03:28,391::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1905::DEBUG::2012-08-30 13:03:29,160::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1905::DEBUG::2012-08-30 13:03:29,160::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::DEBUG::2012-08-30 13:03:30,412::task::588::TaskManager.Task::(_updateState) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:03:30,413::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:03:30,413::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f7f46e10-6291-4d32-9f73-7a31310e7956`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:03:30,413::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:03:30,414::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:03:30,414::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f7f46e10-6291-4d32-9f73-7a31310e7956`::Granted request
>Thread-104::DEBUG::2012-08-30 13:03:30,414::task::817::TaskManager.Task::(resourceAcquired) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:03:30,414::task::978::TaskManager.Task::(_decref) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:03:30,416::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:03:30,418::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:03:30,419::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:03:30,419::task::1172::TaskManager.Task::(prepare) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:03:30,419::task::588::TaskManager.Task::(_updateState) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:03:30,420::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:03:30,420::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:03:30,420::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:03:30,420::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:03:30,421::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:03:30,421::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:03:30,421::task::978::TaskManager.Task::(_decref) Task=`8a8ec8d0-c71d-4210-ad58-a8178c4edade`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:03:30,421::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:30,422::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:30,422::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:30,423::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:03:32,703::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:03:33,392::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:35,424::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1907::DEBUG::2012-08-30 13:03:37,971::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1907::DEBUG::2012-08-30 13:03:37,971::task::588::TaskManager.Task::(_updateState) Task=`9a9bdc34-93d4-4fc8-8f61-5b360e850d85`::moving from state init -> state preparing
>Thread-1907::INFO::2012-08-30 13:03:37,971::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1907::INFO::2012-08-30 13:03:37,972::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1907::DEBUG::2012-08-30 13:03:37,972::task::1172::TaskManager.Task::(prepare) Task=`9a9bdc34-93d4-4fc8-8f61-5b360e850d85`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1907::DEBUG::2012-08-30 13:03:37,972::task::588::TaskManager.Task::(_updateState) Task=`9a9bdc34-93d4-4fc8-8f61-5b360e850d85`::moving from state preparing -> state finished
>Thread-1907::DEBUG::2012-08-30 13:03:37,972::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1907::DEBUG::2012-08-30 13:03:37,972::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1907::DEBUG::2012-08-30 13:03:37,973::task::978::TaskManager.Task::(_decref) Task=`9a9bdc34-93d4-4fc8-8f61-5b360e850d85`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:03:38,393::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:38,394::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:40,425::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1909::DEBUG::2012-08-30 13:03:41,102::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1909::DEBUG::2012-08-30 13:03:41,102::task::588::TaskManager.Task::(_updateState) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::moving from state init -> state preparing
>Thread-1909::INFO::2012-08-30 13:03:41,102::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1909::DEBUG::2012-08-30 13:03:41,103::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3807b644-377c-4da5-a3ff-07d16c0972d2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1909::DEBUG::2012-08-30 13:03:41,103::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1909::DEBUG::2012-08-30 13:03:41,103::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1909::DEBUG::2012-08-30 13:03:41,103::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3807b644-377c-4da5-a3ff-07d16c0972d2`::Granted request
>Thread-1909::DEBUG::2012-08-30 13:03:41,104::task::817::TaskManager.Task::(resourceAcquired) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1909::DEBUG::2012-08-30 13:03:41,104::task::978::TaskManager.Task::(_decref) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::ref 1 aborting False
>Thread-1909::INFO::2012-08-30 13:03:41,107::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1909::DEBUG::2012-08-30 13:03:41,107::task::1172::TaskManager.Task::(prepare) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1909::DEBUG::2012-08-30 13:03:41,107::task::588::TaskManager.Task::(_updateState) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::moving from state preparing -> state finished
>Thread-1909::DEBUG::2012-08-30 13:03:41,108::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1909::DEBUG::2012-08-30 13:03:41,108::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1909::DEBUG::2012-08-30 13:03:41,108::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1909::DEBUG::2012-08-30 13:03:41,108::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1909::DEBUG::2012-08-30 13:03:41,109::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1909::DEBUG::2012-08-30 13:03:41,109::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1909::DEBUG::2012-08-30 13:03:41,109::task::978::TaskManager.Task::(_decref) Task=`fdfde89e-cb71-4ee9-a280-a03b5e40c3ad`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:03:43,395::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:45,426::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:45,426::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:48,396::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:03:50,427::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:03:53,397::task::588::TaskManager.Task::(_updateState) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:03:53,398::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:03:53,398::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`5a856057-1a02-4ecf-919d-340c96127d1a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:03:53,398::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:03:53,398::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:03:53,399::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`5a856057-1a02-4ecf-919d-340c96127d1a`::Granted request
>Thread-40::DEBUG::2012-08-30 13:03:53,399::task::817::TaskManager.Task::(resourceAcquired) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:03:53,399::task::978::TaskManager.Task::(_decref) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:03:53,401::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:03:53,403::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:03:53,404::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:03:53,404::task::1172::TaskManager.Task::(prepare) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:03:53,405::task::588::TaskManager.Task::(_updateState) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:03:53,405::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:03:53,405::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:03:53,405::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:03:53,406::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:03:53,406::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:03:53,406::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:03:53,406::task::978::TaskManager.Task::(_decref) Task=`042c9f0d-784b-4da0-b718-4d7e9c766071`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:03:53,407::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:53,407::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:53,408::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:03:53,408::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1912::DEBUG::2012-08-30 13:03:54,439::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1913::DEBUG::2012-08-30 13:03:54,440::task::588::TaskManager.Task::(_updateState) Task=`9b71abda-beba-4be3-b74b-dd17e2f83288`::moving from state init -> state preparing
>Thread-1912::DEBUG::2012-08-30 13:03:54,440::task::588::TaskManager.Task::(_updateState) Task=`98a2c98e-f450-4c1f-a738-70e14513d2a6`::moving from state init -> state preparing
>Thread-1913::INFO::2012-08-30 13:03:54,440::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1912::INFO::2012-08-30 13:03:54,440::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1913::INFO::2012-08-30 13:03:54,441::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176906585693', 'lastCheck': 1346303030.3632259, 'code': 0, 'valid': True}}
>Thread-1912::INFO::2012-08-30 13:03:54,441::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1913::DEBUG::2012-08-30 13:03:54,441::task::1172::TaskManager.Task::(prepare) Task=`9b71abda-beba-4be3-b74b-dd17e2f83288`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176906585693', 'lastCheck': 1346303030.3632259, 'code': 0, 'valid': True}}
>Thread-1912::DEBUG::2012-08-30 13:03:54,441::task::1172::TaskManager.Task::(prepare) Task=`98a2c98e-f450-4c1f-a738-70e14513d2a6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1913::DEBUG::2012-08-30 13:03:54,442::task::588::TaskManager.Task::(_updateState) Task=`9b71abda-beba-4be3-b74b-dd17e2f83288`::moving from state preparing -> state finished
>Thread-1912::DEBUG::2012-08-30 13:03:54,442::task::588::TaskManager.Task::(_updateState) Task=`98a2c98e-f450-4c1f-a738-70e14513d2a6`::moving from state preparing -> state finished
>Thread-1913::DEBUG::2012-08-30 13:03:54,442::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1912::DEBUG::2012-08-30 13:03:54,442::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1913::DEBUG::2012-08-30 13:03:54,442::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1912::DEBUG::2012-08-30 13:03:54,442::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1913::DEBUG::2012-08-30 13:03:54,443::task::978::TaskManager.Task::(_decref) Task=`9b71abda-beba-4be3-b74b-dd17e2f83288`::ref 0 aborting False
>Thread-1912::DEBUG::2012-08-30 13:03:54,443::task::978::TaskManager.Task::(_decref) Task=`98a2c98e-f450-4c1f-a738-70e14513d2a6`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:03:55,428::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:03:55,723::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1914::DEBUG::2012-08-30 13:03:57,596::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1914::DEBUG::2012-08-30 13:03:57,596::task::588::TaskManager.Task::(_updateState) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::moving from state init -> state preparing
>Thread-1914::INFO::2012-08-30 13:03:57,596::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1914::DEBUG::2012-08-30 13:03:57,597::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0cc7179-ade4-4f5d-9024-4c0459482ca5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1914::DEBUG::2012-08-30 13:03:57,597::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1914::DEBUG::2012-08-30 13:03:57,597::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1914::DEBUG::2012-08-30 13:03:57,597::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0cc7179-ade4-4f5d-9024-4c0459482ca5`::Granted request
>Thread-1914::DEBUG::2012-08-30 13:03:57,598::task::817::TaskManager.Task::(resourceAcquired) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1914::DEBUG::2012-08-30 13:03:57,598::task::978::TaskManager.Task::(_decref) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::ref 1 aborting False
>Thread-1915::DEBUG::2012-08-30 13:03:57,599::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1915::DEBUG::2012-08-30 13:03:57,599::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1914::INFO::2012-08-30 13:03:57,603::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1914::DEBUG::2012-08-30 13:03:57,603::task::1172::TaskManager.Task::(prepare) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1914::DEBUG::2012-08-30 13:03:57,604::task::588::TaskManager.Task::(_updateState) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::moving from state preparing -> state finished
>Thread-1914::DEBUG::2012-08-30 13:03:57,604::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1914::DEBUG::2012-08-30 13:03:57,604::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1914::DEBUG::2012-08-30 13:03:57,605::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1914::DEBUG::2012-08-30 13:03:57,605::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1914::DEBUG::2012-08-30 13:03:57,605::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1914::DEBUG::2012-08-30 13:03:57,605::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1914::DEBUG::2012-08-30 13:03:57,605::task::978::TaskManager.Task::(_decref) Task=`cd00eee3-0b59-46aa-ac55-dbf4773cce03`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:03:58,409::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:00,429::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:00,429::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:04:02,730::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:04:03,410::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:05,430::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:08,412::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:08,412::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:10,431::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1918::DEBUG::2012-08-30 13:04:10,816::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1918::DEBUG::2012-08-30 13:04:10,816::task::588::TaskManager.Task::(_updateState) Task=`4ce3b2c5-d4fc-4ed6-9eaf-546e86bd2ee2`::moving from state init -> state preparing
>Thread-1918::INFO::2012-08-30 13:04:10,817::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1918::INFO::2012-08-30 13:04:10,817::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1918::DEBUG::2012-08-30 13:04:10,817::task::1172::TaskManager.Task::(prepare) Task=`4ce3b2c5-d4fc-4ed6-9eaf-546e86bd2ee2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1918::DEBUG::2012-08-30 13:04:10,817::task::588::TaskManager.Task::(_updateState) Task=`4ce3b2c5-d4fc-4ed6-9eaf-546e86bd2ee2`::moving from state preparing -> state finished
>Thread-1918::DEBUG::2012-08-30 13:04:10,818::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1918::DEBUG::2012-08-30 13:04:10,818::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1918::DEBUG::2012-08-30 13:04:10,818::task::978::TaskManager.Task::(_decref) Task=`4ce3b2c5-d4fc-4ed6-9eaf-546e86bd2ee2`::ref 0 aborting False
>Thread-1920::DEBUG::2012-08-30 13:04:13,349::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1920::DEBUG::2012-08-30 13:04:13,350::task::588::TaskManager.Task::(_updateState) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::moving from state init -> state preparing
>Thread-1920::INFO::2012-08-30 13:04:13,351::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1920::DEBUG::2012-08-30 13:04:13,351::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5f4972ec-1136-4872-b821-01441689ec19`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1920::DEBUG::2012-08-30 13:04:13,351::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1920::DEBUG::2012-08-30 13:04:13,351::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1920::DEBUG::2012-08-30 13:04:13,352::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5f4972ec-1136-4872-b821-01441689ec19`::Granted request
>Thread-1920::DEBUG::2012-08-30 13:04:13,352::task::817::TaskManager.Task::(resourceAcquired) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1920::DEBUG::2012-08-30 13:04:13,352::task::978::TaskManager.Task::(_decref) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::ref 1 aborting False
>Thread-1920::INFO::2012-08-30 13:04:13,355::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1920::DEBUG::2012-08-30 13:04:13,355::task::1172::TaskManager.Task::(prepare) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1920::DEBUG::2012-08-30 13:04:13,356::task::588::TaskManager.Task::(_updateState) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::moving from state preparing -> state finished
>Thread-1920::DEBUG::2012-08-30 13:04:13,356::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1920::DEBUG::2012-08-30 13:04:13,356::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1920::DEBUG::2012-08-30 13:04:13,356::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1920::DEBUG::2012-08-30 13:04:13,357::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1920::DEBUG::2012-08-30 13:04:13,357::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1920::DEBUG::2012-08-30 13:04:13,357::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1920::DEBUG::2012-08-30 13:04:13,357::task::978::TaskManager.Task::(_decref) Task=`4ff13cf0-d8c7-41a6-9182-58e71ac89ed7`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:04:13,413::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:15,432::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:15,433::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:18,414::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:20,433::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:23,416::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:23,416::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1922::DEBUG::2012-08-30 13:04:23,821::task::588::TaskManager.Task::(_updateState) Task=`077ded06-397b-41e8-844d-44fc518841d1`::moving from state init -> state preparing
>Thread-1922::INFO::2012-08-30 13:04:23,822::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1922::INFO::2012-08-30 13:04:23,822::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162315368652', 'lastCheck': 1346303060.383126, 'code': 0, 'valid': True}}
>Thread-1922::DEBUG::2012-08-30 13:04:23,822::task::1172::TaskManager.Task::(prepare) Task=`077ded06-397b-41e8-844d-44fc518841d1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162315368652', 'lastCheck': 1346303060.383126, 'code': 0, 'valid': True}}
>Thread-1922::DEBUG::2012-08-30 13:04:23,822::task::588::TaskManager.Task::(_updateState) Task=`077ded06-397b-41e8-844d-44fc518841d1`::moving from state preparing -> state finished
>Thread-1922::DEBUG::2012-08-30 13:04:23,823::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1922::DEBUG::2012-08-30 13:04:23,823::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1922::DEBUG::2012-08-30 13:04:23,823::task::978::TaskManager.Task::(_decref) Task=`077ded06-397b-41e8-844d-44fc518841d1`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:04:25,434::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:04:25,750::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1924::DEBUG::2012-08-30 13:04:26,627::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1923::DEBUG::2012-08-30 13:04:26,627::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1924::DEBUG::2012-08-30 13:04:26,627::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-1923::DEBUG::2012-08-30 13:04:26,628::task::588::TaskManager.Task::(_updateState) Task=`9cc5f57e-d8a7-488f-90c6-d50e3eb0a097`::moving from state init -> state preparing
>Thread-1923::INFO::2012-08-30 13:04:26,629::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1923::INFO::2012-08-30 13:04:26,629::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1923::DEBUG::2012-08-30 13:04:26,630::task::1172::TaskManager.Task::(prepare) Task=`9cc5f57e-d8a7-488f-90c6-d50e3eb0a097`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1923::DEBUG::2012-08-30 13:04:26,630::task::588::TaskManager.Task::(_updateState) Task=`9cc5f57e-d8a7-488f-90c6-d50e3eb0a097`::moving from state preparing -> state finished
>Thread-1923::DEBUG::2012-08-30 13:04:26,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1923::DEBUG::2012-08-30 13:04:26,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1923::DEBUG::2012-08-30 13:04:26,631::task::978::TaskManager.Task::(_decref) Task=`9cc5f57e-d8a7-488f-90c6-d50e3eb0a097`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:04:28,417::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1925::DEBUG::2012-08-30 13:04:29,783::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1925::DEBUG::2012-08-30 13:04:29,784::task::588::TaskManager.Task::(_updateState) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::moving from state init -> state preparing
>Thread-1925::INFO::2012-08-30 13:04:29,784::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1925::DEBUG::2012-08-30 13:04:29,784::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`af8e09dc-d20f-4f9f-8d2f-9785c56aa197`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1925::DEBUG::2012-08-30 13:04:29,785::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1925::DEBUG::2012-08-30 13:04:29,785::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1925::DEBUG::2012-08-30 13:04:29,785::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`af8e09dc-d20f-4f9f-8d2f-9785c56aa197`::Granted request
>Thread-1925::DEBUG::2012-08-30 13:04:29,785::task::817::TaskManager.Task::(resourceAcquired) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1925::DEBUG::2012-08-30 13:04:29,786::task::978::TaskManager.Task::(_decref) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::ref 1 aborting False
>Thread-1925::INFO::2012-08-30 13:04:29,789::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1925::DEBUG::2012-08-30 13:04:29,789::task::1172::TaskManager.Task::(prepare) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1925::DEBUG::2012-08-30 13:04:29,789::task::588::TaskManager.Task::(_updateState) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::moving from state preparing -> state finished
>Thread-1925::DEBUG::2012-08-30 13:04:29,789::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1925::DEBUG::2012-08-30 13:04:29,790::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1925::DEBUG::2012-08-30 13:04:29,791::task::978::TaskManager.Task::(_decref) Task=`ba773b5a-166e-4405-bf08-0d9fd87cf531`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 13:04:30,435::task::588::TaskManager.Task::(_updateState) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:04:30,436::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:04:30,436::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`15e13fd9-75d4-439c-889a-7de5d22808a4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:04:30,436::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:04:30,437::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:04:30,437::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`15e13fd9-75d4-439c-889a-7de5d22808a4`::Granted request
>Thread-104::DEBUG::2012-08-30 13:04:30,437::task::817::TaskManager.Task::(resourceAcquired) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:04:30,437::task::978::TaskManager.Task::(_decref) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:04:30,439::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:04:30,441::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:04:30,442::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:04:30,442::task::1172::TaskManager.Task::(prepare) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:04:30,442::task::588::TaskManager.Task::(_updateState) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:04:30,443::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:04:30,443::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:04:30,443::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:04:30,443::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:04:30,444::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:04:30,444::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:04:30,444::task::978::TaskManager.Task::(_decref) Task=`c146bcd0-aad5-4b8a-877b-cd405a20ebe0`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:04:30,445::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:30,445::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:30,445::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:30,446::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:04:32,757::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:04:33,418::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:35,447::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:38,419::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:38,420::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:40,448::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1929::DEBUG::2012-08-30 13:04:42,985::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1929::DEBUG::2012-08-30 13:04:42,986::task::588::TaskManager.Task::(_updateState) Task=`b5c3068e-220a-40cf-be44-6f1b4da94127`::moving from state init -> state preparing
>Thread-1929::INFO::2012-08-30 13:04:42,986::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1929::INFO::2012-08-30 13:04:42,986::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1929::DEBUG::2012-08-30 13:04:42,986::task::1172::TaskManager.Task::(prepare) Task=`b5c3068e-220a-40cf-be44-6f1b4da94127`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1929::DEBUG::2012-08-30 13:04:42,987::task::588::TaskManager.Task::(_updateState) Task=`b5c3068e-220a-40cf-be44-6f1b4da94127`::moving from state preparing -> state finished
>Thread-1929::DEBUG::2012-08-30 13:04:42,987::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1929::DEBUG::2012-08-30 13:04:42,987::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1929::DEBUG::2012-08-30 13:04:42,987::task::978::TaskManager.Task::(_decref) Task=`b5c3068e-220a-40cf-be44-6f1b4da94127`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:04:43,421::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:45,449::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:45,450::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1930::DEBUG::2012-08-30 13:04:46,125::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1930::DEBUG::2012-08-30 13:04:46,126::task::588::TaskManager.Task::(_updateState) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::moving from state init -> state preparing
>Thread-1930::INFO::2012-08-30 13:04:46,126::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1930::DEBUG::2012-08-30 13:04:46,126::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0cdb43c9-e7a5-4761-ba31-9f46e0767443`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1930::DEBUG::2012-08-30 13:04:46,126::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1930::DEBUG::2012-08-30 13:04:46,127::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1930::DEBUG::2012-08-30 13:04:46,127::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0cdb43c9-e7a5-4761-ba31-9f46e0767443`::Granted request
>Thread-1930::DEBUG::2012-08-30 13:04:46,127::task::817::TaskManager.Task::(resourceAcquired) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1930::DEBUG::2012-08-30 13:04:46,128::task::978::TaskManager.Task::(_decref) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::ref 1 aborting False
>Thread-1930::INFO::2012-08-30 13:04:46,130::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1930::DEBUG::2012-08-30 13:04:46,131::task::1172::TaskManager.Task::(prepare) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1930::DEBUG::2012-08-30 13:04:46,131::task::588::TaskManager.Task::(_updateState) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::moving from state preparing -> state finished
>Thread-1930::DEBUG::2012-08-30 13:04:46,131::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1930::DEBUG::2012-08-30 13:04:46,131::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1930::DEBUG::2012-08-30 13:04:46,132::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1930::DEBUG::2012-08-30 13:04:46,132::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1930::DEBUG::2012-08-30 13:04:46,132::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1930::DEBUG::2012-08-30 13:04:46,132::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1930::DEBUG::2012-08-30 13:04:46,133::task::978::TaskManager.Task::(_decref) Task=`59ed1bd1-4ef6-4943-a646-77bbb4104a4b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:04:48,421::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:04:50,451::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1932::DEBUG::2012-08-30 13:04:51,273::task::588::TaskManager.Task::(_updateState) Task=`ed585a9a-7fec-4320-a43c-1d1345a802d8`::moving from state init -> state preparing
>Thread-1932::INFO::2012-08-30 13:04:51,273::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1932::INFO::2012-08-30 13:04:51,273::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177884101868', 'lastCheck': 1346303090.4032381, 'code': 0, 'valid': True}}
>Thread-1932::DEBUG::2012-08-30 13:04:51,274::task::1172::TaskManager.Task::(prepare) Task=`ed585a9a-7fec-4320-a43c-1d1345a802d8`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177884101868', 'lastCheck': 1346303090.4032381, 'code': 0, 'valid': True}}
>Thread-1932::DEBUG::2012-08-30 13:04:51,274::task::588::TaskManager.Task::(_updateState) Task=`ed585a9a-7fec-4320-a43c-1d1345a802d8`::moving from state preparing -> state finished
>Thread-1932::DEBUG::2012-08-30 13:04:51,274::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1932::DEBUG::2012-08-30 13:04:51,274::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1932::DEBUG::2012-08-30 13:04:51,274::task::978::TaskManager.Task::(_decref) Task=`ed585a9a-7fec-4320-a43c-1d1345a802d8`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 13:04:53,422::task::588::TaskManager.Task::(_updateState) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:04:53,423::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:04:53,423::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ccf26099-948f-4506-ae65-3c708e974e34`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:04:53,423::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:04:53,424::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:04:53,424::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ccf26099-948f-4506-ae65-3c708e974e34`::Granted request
>Thread-40::DEBUG::2012-08-30 13:04:53,424::task::817::TaskManager.Task::(resourceAcquired) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:04:53,425::task::978::TaskManager.Task::(_decref) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:04:53,426::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:04:53,428::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:04:53,430::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:04:53,430::task::1172::TaskManager.Task::(prepare) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:04:53,430::task::588::TaskManager.Task::(_updateState) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:04:53,430::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:04:53,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:04:53,431::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:04:53,431::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:04:53,431::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:04:53,431::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:04:53,432::task::978::TaskManager.Task::(_decref) Task=`9e09a74a-2081-4f25-9a01-a875ec43d40a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:04:53,432::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:53,433::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:53,433::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:04:53,433::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1933::DEBUG::2012-08-30 13:04:54,415::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1933::DEBUG::2012-08-30 13:04:54,415::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:04:55,452::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:04:55,775::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:04:58,434::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1934::DEBUG::2012-08-30 13:04:59,352::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1934::DEBUG::2012-08-30 13:04:59,352::task::588::TaskManager.Task::(_updateState) Task=`21395c7c-c90e-4965-8416-26950534b187`::moving from state init -> state preparing
>Thread-1934::INFO::2012-08-30 13:04:59,352::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1934::INFO::2012-08-30 13:04:59,353::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1934::DEBUG::2012-08-30 13:04:59,353::task::1172::TaskManager.Task::(prepare) Task=`21395c7c-c90e-4965-8416-26950534b187`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1934::DEBUG::2012-08-30 13:04:59,353::task::588::TaskManager.Task::(_updateState) Task=`21395c7c-c90e-4965-8416-26950534b187`::moving from state preparing -> state finished
>Thread-1934::DEBUG::2012-08-30 13:04:59,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1934::DEBUG::2012-08-30 13:04:59,354::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1934::DEBUG::2012-08-30 13:04:59,354::task::978::TaskManager.Task::(_decref) Task=`21395c7c-c90e-4965-8416-26950534b187`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:05:00,454::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:00,454::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1936::DEBUG::2012-08-30 13:05:02,492::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1936::DEBUG::2012-08-30 13:05:02,492::task::588::TaskManager.Task::(_updateState) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::moving from state init -> state preparing
>Thread-1936::INFO::2012-08-30 13:05:02,493::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1936::DEBUG::2012-08-30 13:05:02,493::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2c81b8e6-2905-4d49-9bf8-abca1bad3fe9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1936::DEBUG::2012-08-30 13:05:02,493::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1936::DEBUG::2012-08-30 13:05:02,494::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1936::DEBUG::2012-08-30 13:05:02,494::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2c81b8e6-2905-4d49-9bf8-abca1bad3fe9`::Granted request
>Thread-1936::DEBUG::2012-08-30 13:05:02,495::task::817::TaskManager.Task::(resourceAcquired) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1936::DEBUG::2012-08-30 13:05:02,495::task::978::TaskManager.Task::(_decref) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::ref 1 aborting False
>Thread-1936::INFO::2012-08-30 13:05:02,498::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1936::DEBUG::2012-08-30 13:05:02,498::task::1172::TaskManager.Task::(prepare) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1936::DEBUG::2012-08-30 13:05:02,498::task::588::TaskManager.Task::(_updateState) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::moving from state preparing -> state finished
>Thread-1936::DEBUG::2012-08-30 13:05:02,498::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1936::DEBUG::2012-08-30 13:05:02,499::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1936::DEBUG::2012-08-30 13:05:02,499::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1936::DEBUG::2012-08-30 13:05:02,499::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1936::DEBUG::2012-08-30 13:05:02,499::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1936::DEBUG::2012-08-30 13:05:02,500::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1936::DEBUG::2012-08-30 13:05:02,500::task::978::TaskManager.Task::(_decref) Task=`0686b056-53dd-47e4-b970-1ddf0101c78f`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:05:02,779::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:05:03,435::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:05,455::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:08,436::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:08,437::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:10,456::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:13,438::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:15,458::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:15,458::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1940::DEBUG::2012-08-30 13:05:15,680::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1940::DEBUG::2012-08-30 13:05:15,680::task::588::TaskManager.Task::(_updateState) Task=`50b5a93f-eeb3-4bd4-8463-2e6fb0822766`::moving from state init -> state preparing
>Thread-1940::INFO::2012-08-30 13:05:15,680::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1940::INFO::2012-08-30 13:05:15,681::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1940::DEBUG::2012-08-30 13:05:15,681::task::1172::TaskManager.Task::(prepare) Task=`50b5a93f-eeb3-4bd4-8463-2e6fb0822766`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1940::DEBUG::2012-08-30 13:05:15,681::task::588::TaskManager.Task::(_updateState) Task=`50b5a93f-eeb3-4bd4-8463-2e6fb0822766`::moving from state preparing -> state finished
>Thread-1940::DEBUG::2012-08-30 13:05:15,681::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1940::DEBUG::2012-08-30 13:05:15,682::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1940::DEBUG::2012-08-30 13:05:15,682::task::978::TaskManager.Task::(_decref) Task=`50b5a93f-eeb3-4bd4-8463-2e6fb0822766`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:05:18,439::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1941::DEBUG::2012-08-30 13:05:18,820::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1942::DEBUG::2012-08-30 13:05:18,821::task::588::TaskManager.Task::(_updateState) Task=`f5d36a44-3d9d-4ca4-8330-e04024f6fec4`::moving from state init -> state preparing
>Thread-1941::DEBUG::2012-08-30 13:05:18,821::task::588::TaskManager.Task::(_updateState) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::moving from state init -> state preparing
>Thread-1942::INFO::2012-08-30 13:05:18,821::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1941::INFO::2012-08-30 13:05:18,821::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1942::INFO::2012-08-30 13:05:18,822::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00266218185425', 'lastCheck': 1346303110.418643, 'code': 0, 'valid': True}}
>Thread-1941::DEBUG::2012-08-30 13:05:18,822::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`76cf2168-d497-4f1d-92b6-7f3fc519d000`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1942::DEBUG::2012-08-30 13:05:18,822::task::1172::TaskManager.Task::(prepare) Task=`f5d36a44-3d9d-4ca4-8330-e04024f6fec4`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00266218185425', 'lastCheck': 1346303110.418643, 'code': 0, 'valid': True}}
>Thread-1941::DEBUG::2012-08-30 13:05:18,822::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1942::DEBUG::2012-08-30 13:05:18,823::task::588::TaskManager.Task::(_updateState) Task=`f5d36a44-3d9d-4ca4-8330-e04024f6fec4`::moving from state preparing -> state finished
>Thread-1941::DEBUG::2012-08-30 13:05:18,823::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1942::DEBUG::2012-08-30 13:05:18,823::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1941::DEBUG::2012-08-30 13:05:18,823::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`76cf2168-d497-4f1d-92b6-7f3fc519d000`::Granted request
>Thread-1942::DEBUG::2012-08-30 13:05:18,823::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1941::DEBUG::2012-08-30 13:05:18,824::task::817::TaskManager.Task::(resourceAcquired) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1942::DEBUG::2012-08-30 13:05:18,824::task::978::TaskManager.Task::(_decref) Task=`f5d36a44-3d9d-4ca4-8330-e04024f6fec4`::ref 0 aborting False
>Thread-1941::DEBUG::2012-08-30 13:05:18,824::task::978::TaskManager.Task::(_decref) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::ref 1 aborting False
>Thread-1941::INFO::2012-08-30 13:05:18,829::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1941::DEBUG::2012-08-30 13:05:18,829::task::1172::TaskManager.Task::(prepare) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1941::DEBUG::2012-08-30 13:05:18,829::task::588::TaskManager.Task::(_updateState) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::moving from state preparing -> state finished
>Thread-1941::DEBUG::2012-08-30 13:05:18,830::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1941::DEBUG::2012-08-30 13:05:18,830::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1941::DEBUG::2012-08-30 13:05:18,830::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1941::DEBUG::2012-08-30 13:05:18,830::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1941::DEBUG::2012-08-30 13:05:18,831::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1941::DEBUG::2012-08-30 13:05:18,831::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1941::DEBUG::2012-08-30 13:05:18,831::task::978::TaskManager.Task::(_decref) Task=`0877052c-3093-4bf3-a7d4-50c2ab95e9a2`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:05:20,459::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1943::DEBUG::2012-08-30 13:05:21,966::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1943::DEBUG::2012-08-30 13:05:21,967::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:05:23,440::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:23,440::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:25,461::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:05:25,798::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:05:28,441::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-26::DEBUG::2012-08-30 13:05:30,426::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:05:30,426::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-26::DEBUG::2012-08-30 13:05:30,427::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None)
>Thread-104::DEBUG::2012-08-30 13:05:30,462::task::588::TaskManager.Task::(_updateState) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:05:30,463::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:05:30,463::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`216b723a-03a8-424c-bec4-1091afa4c5d5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:05:30,463::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:05:30,464::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:05:30,464::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`216b723a-03a8-424c-bec4-1091afa4c5d5`::Granted request
>Thread-104::DEBUG::2012-08-30 13:05:30,464::task::817::TaskManager.Task::(resourceAcquired) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:05:30,465::task::978::TaskManager.Task::(_decref) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::ref 1 aborting False
>Thread-26::DEBUG::2012-08-30 13:05:30,541::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = '  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5
>Thread-26::WARNING::2012-08-30 13:05:30,541::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found']
>Thread-26::DEBUG::2012-08-30 13:05:30,542::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-26::DEBUG::2012-08-30 13:05:30,548::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:05:30,548::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend
>Thread-26::DEBUG::2012-08-30 13:05:30,550::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d']
>Thread-26::DEBUG::2012-08-30 13:05:30,552::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-26::WARNING::2012-08-30 13:05:30,552::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered
>Thread-26::WARNING::2012-08-30 13:05:30,552::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered
>Thread-104::DEBUG::2012-08-30 13:05:30,554::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:05:30,564::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:05:30,566::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:05:30,566::task::1172::TaskManager.Task::(prepare) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:05:30,566::task::588::TaskManager.Task::(_updateState) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:05:30,567::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:05:30,567::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:05:30,567::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:05:30,568::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:05:30,568::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:05:30,568::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:05:30,568::task::978::TaskManager.Task::(_decref) Task=`9cd595d6-0a19-420e-b376-2e64a61a151d`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:05:30,569::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:30,569::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:30,570::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:30,570::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1945::DEBUG::2012-08-30 13:05:32,023::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1945::DEBUG::2012-08-30 13:05:32,024::task::588::TaskManager.Task::(_updateState) Task=`67dd5280-d24f-491c-b08f-ac2ad2853152`::moving from state init -> state preparing
>Thread-1945::INFO::2012-08-30 13:05:32,024::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1945::INFO::2012-08-30 13:05:32,024::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1945::DEBUG::2012-08-30 13:05:32,025::task::1172::TaskManager.Task::(prepare) Task=`67dd5280-d24f-491c-b08f-ac2ad2853152`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1945::DEBUG::2012-08-30 13:05:32,025::task::588::TaskManager.Task::(_updateState) Task=`67dd5280-d24f-491c-b08f-ac2ad2853152`::moving from state preparing -> state finished
>Thread-1945::DEBUG::2012-08-30 13:05:32,025::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1945::DEBUG::2012-08-30 13:05:32,025::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1945::DEBUG::2012-08-30 13:05:32,025::task::978::TaskManager.Task::(_decref) Task=`67dd5280-d24f-491c-b08f-ac2ad2853152`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:05:32,805::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:05:33,442::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1952::DEBUG::2012-08-30 13:05:35,312::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1952::DEBUG::2012-08-30 13:05:35,312::task::588::TaskManager.Task::(_updateState) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::moving from state init -> state preparing
>Thread-1952::INFO::2012-08-30 13:05:35,312::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1952::DEBUG::2012-08-30 13:05:35,313::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f6ad3412-0190-4815-9886-f8e13d60d9b2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1952::DEBUG::2012-08-30 13:05:35,313::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1952::DEBUG::2012-08-30 13:05:35,313::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1952::DEBUG::2012-08-30 13:05:35,313::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f6ad3412-0190-4815-9886-f8e13d60d9b2`::Granted request
>Thread-1952::DEBUG::2012-08-30 13:05:35,314::task::817::TaskManager.Task::(resourceAcquired) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1952::DEBUG::2012-08-30 13:05:35,314::task::978::TaskManager.Task::(_decref) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::ref 1 aborting False
>Thread-1952::INFO::2012-08-30 13:05:35,317::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1952::DEBUG::2012-08-30 13:05:35,317::task::1172::TaskManager.Task::(prepare) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1952::DEBUG::2012-08-30 13:05:35,318::task::588::TaskManager.Task::(_updateState) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::moving from state preparing -> state finished
>Thread-1952::DEBUG::2012-08-30 13:05:35,318::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1952::DEBUG::2012-08-30 13:05:35,318::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1952::DEBUG::2012-08-30 13:05:35,318::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1952::DEBUG::2012-08-30 13:05:35,319::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1952::DEBUG::2012-08-30 13:05:35,319::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1952::DEBUG::2012-08-30 13:05:35,319::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1952::DEBUG::2012-08-30 13:05:35,319::task::978::TaskManager.Task::(_decref) Task=`262779e5-ff8f-4a81-b339-ad9c8455ab8b`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:05:35,572::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:38,443::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:38,444::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:40,573::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:43,444::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:45,574::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:45,575::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1955::DEBUG::2012-08-30 13:05:46,989::task::588::TaskManager.Task::(_updateState) Task=`1b46c342-9527-41b6-8fb5-6140cdd45304`::moving from state init -> state preparing
>Thread-1955::INFO::2012-08-30 13:05:46,990::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1955::INFO::2012-08-30 13:05:46,990::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303140.610781, 'code': 0, 'valid': True}}
>Thread-1955::DEBUG::2012-08-30 13:05:46,990::task::1172::TaskManager.Task::(prepare) Task=`1b46c342-9527-41b6-8fb5-6140cdd45304`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303140.610781, 'code': 0, 'valid': True}}
>Thread-1955::DEBUG::2012-08-30 13:05:46,991::task::588::TaskManager.Task::(_updateState) Task=`1b46c342-9527-41b6-8fb5-6140cdd45304`::moving from state preparing -> state finished
>Thread-1955::DEBUG::2012-08-30 13:05:46,991::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1955::DEBUG::2012-08-30 13:05:46,991::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1955::DEBUG::2012-08-30 13:05:46,991::task::978::TaskManager.Task::(_decref) Task=`1b46c342-9527-41b6-8fb5-6140cdd45304`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:05:48,445::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1956::DEBUG::2012-08-30 13:05:48,564::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1956::DEBUG::2012-08-30 13:05:48,565::task::588::TaskManager.Task::(_updateState) Task=`3394e19f-2c77-461c-bd06-bbcdf348c612`::moving from state init -> state preparing
>Thread-1956::INFO::2012-08-30 13:05:48,565::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1956::INFO::2012-08-30 13:05:48,565::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1956::DEBUG::2012-08-30 13:05:48,566::task::1172::TaskManager.Task::(prepare) Task=`3394e19f-2c77-461c-bd06-bbcdf348c612`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1956::DEBUG::2012-08-30 13:05:48,566::task::588::TaskManager.Task::(_updateState) Task=`3394e19f-2c77-461c-bd06-bbcdf348c612`::moving from state preparing -> state finished
>Thread-1956::DEBUG::2012-08-30 13:05:48,566::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1956::DEBUG::2012-08-30 13:05:48,566::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1956::DEBUG::2012-08-30 13:05:48,566::task::978::TaskManager.Task::(_decref) Task=`3394e19f-2c77-461c-bd06-bbcdf348c612`::ref 0 aborting False
>Thread-1957::DEBUG::2012-08-30 13:05:50,139::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1957::DEBUG::2012-08-30 13:05:50,140::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:05:50,576::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1958::DEBUG::2012-08-30 13:05:51,712::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1958::DEBUG::2012-08-30 13:05:51,713::task::588::TaskManager.Task::(_updateState) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::moving from state init -> state preparing
>Thread-1958::INFO::2012-08-30 13:05:51,713::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1958::DEBUG::2012-08-30 13:05:51,713::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e44b5af8-850a-40f8-8bae-ca6c94d3e8f2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1958::DEBUG::2012-08-30 13:05:51,714::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1958::DEBUG::2012-08-30 13:05:51,714::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1958::DEBUG::2012-08-30 13:05:51,714::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e44b5af8-850a-40f8-8bae-ca6c94d3e8f2`::Granted request
>Thread-1958::DEBUG::2012-08-30 13:05:51,714::task::817::TaskManager.Task::(resourceAcquired) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1958::DEBUG::2012-08-30 13:05:51,715::task::978::TaskManager.Task::(_decref) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::ref 1 aborting False
>Thread-1958::INFO::2012-08-30 13:05:51,718::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1958::DEBUG::2012-08-30 13:05:51,718::task::1172::TaskManager.Task::(prepare) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1958::DEBUG::2012-08-30 13:05:51,718::task::588::TaskManager.Task::(_updateState) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::moving from state preparing -> state finished
>Thread-1958::DEBUG::2012-08-30 13:05:51,718::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1958::DEBUG::2012-08-30 13:05:51,718::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1958::DEBUG::2012-08-30 13:05:51,719::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1958::DEBUG::2012-08-30 13:05:51,719::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1958::DEBUG::2012-08-30 13:05:51,719::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1958::DEBUG::2012-08-30 13:05:51,719::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1958::DEBUG::2012-08-30 13:05:51,720::task::978::TaskManager.Task::(_decref) Task=`e3292211-e10b-4748-bcae-9ffcb9209304`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 13:05:53,446::task::588::TaskManager.Task::(_updateState) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:05:53,447::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:05:53,447::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f1892dc5-144e-4c10-bdbf-cbacd35db739`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:05:53,447::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:05:53,448::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:05:53,448::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f1892dc5-144e-4c10-bdbf-cbacd35db739`::Granted request
>Thread-40::DEBUG::2012-08-30 13:05:53,448::task::817::TaskManager.Task::(resourceAcquired) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:05:53,448::task::978::TaskManager.Task::(_decref) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:05:53,450::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:05:53,452::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:05:53,453::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:05:53,454::task::1172::TaskManager.Task::(prepare) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:05:53,454::task::588::TaskManager.Task::(_updateState) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:05:53,454::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:05:53,454::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:05:53,455::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:05:53,455::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:05:53,455::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:05:53,455::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:05:53,456::task::978::TaskManager.Task::(_decref) Task=`4e79b3e4-976b-48e0-b4e2-62a8cc89a71f`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:05:53,456::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:53,456::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:53,457::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:05:53,457::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:05:55,577::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:05:55,824::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:05:58,458::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:00,578::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:00,579::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:06:02,829::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:06:03,459::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:05,580::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1961::DEBUG::2012-08-30 13:06:06,373::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1961::DEBUG::2012-08-30 13:06:06,374::task::588::TaskManager.Task::(_updateState) Task=`866af8c6-c747-4add-9634-ceb69fc447bf`::moving from state init -> state preparing
>Thread-1961::INFO::2012-08-30 13:06:06,374::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1961::INFO::2012-08-30 13:06:06,374::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1961::DEBUG::2012-08-30 13:06:06,374::task::1172::TaskManager.Task::(prepare) Task=`866af8c6-c747-4add-9634-ceb69fc447bf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1961::DEBUG::2012-08-30 13:06:06,375::task::588::TaskManager.Task::(_updateState) Task=`866af8c6-c747-4add-9634-ceb69fc447bf`::moving from state preparing -> state finished
>Thread-1961::DEBUG::2012-08-30 13:06:06,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1961::DEBUG::2012-08-30 13:06:06,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1961::DEBUG::2012-08-30 13:06:06,375::task::978::TaskManager.Task::(_decref) Task=`866af8c6-c747-4add-9634-ceb69fc447bf`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:06:08,460::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:08,460::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1963::DEBUG::2012-08-30 13:06:09,562::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1963::DEBUG::2012-08-30 13:06:09,562::task::588::TaskManager.Task::(_updateState) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::moving from state init -> state preparing
>Thread-1963::INFO::2012-08-30 13:06:09,563::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1963::DEBUG::2012-08-30 13:06:09,563::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8977a339-343f-47af-ba27-544ee7c79a4e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1963::DEBUG::2012-08-30 13:06:09,563::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1963::DEBUG::2012-08-30 13:06:09,563::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1963::DEBUG::2012-08-30 13:06:09,564::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`8977a339-343f-47af-ba27-544ee7c79a4e`::Granted request
>Thread-1963::DEBUG::2012-08-30 13:06:09,564::task::817::TaskManager.Task::(resourceAcquired) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1963::DEBUG::2012-08-30 13:06:09,564::task::978::TaskManager.Task::(_decref) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::ref 1 aborting False
>Thread-1963::INFO::2012-08-30 13:06:09,567::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1963::DEBUG::2012-08-30 13:06:09,567::task::1172::TaskManager.Task::(prepare) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1963::DEBUG::2012-08-30 13:06:09,568::task::588::TaskManager.Task::(_updateState) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::moving from state preparing -> state finished
>Thread-1963::DEBUG::2012-08-30 13:06:09,568::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1963::DEBUG::2012-08-30 13:06:09,568::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1963::DEBUG::2012-08-30 13:06:09,568::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1963::DEBUG::2012-08-30 13:06:09,569::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1963::DEBUG::2012-08-30 13:06:09,569::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1963::DEBUG::2012-08-30 13:06:09,569::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1963::DEBUG::2012-08-30 13:06:09,569::task::978::TaskManager.Task::(_decref) Task=`fb4ef3df-801a-4cb2-ab3c-51e999df1b12`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:06:10,581::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:13,461::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:15,582::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:15,583::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1965::DEBUG::2012-08-30 13:06:17,545::task::588::TaskManager.Task::(_updateState) Task=`b53d6443-efe4-4af1-9d45-50a7c90f6000`::moving from state init -> state preparing
>Thread-1965::INFO::2012-08-30 13:06:17,545::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1965::INFO::2012-08-30 13:06:17,546::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303170.631319, 'code': 0, 'valid': True}}
>Thread-1965::DEBUG::2012-08-30 13:06:17,546::task::1172::TaskManager.Task::(prepare) Task=`b53d6443-efe4-4af1-9d45-50a7c90f6000`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303170.631319, 'code': 0, 'valid': True}}
>Thread-1965::DEBUG::2012-08-30 13:06:17,546::task::588::TaskManager.Task::(_updateState) Task=`b53d6443-efe4-4af1-9d45-50a7c90f6000`::moving from state preparing -> state finished
>Thread-1965::DEBUG::2012-08-30 13:06:17,546::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1965::DEBUG::2012-08-30 13:06:17,547::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1965::DEBUG::2012-08-30 13:06:17,547::task::978::TaskManager.Task::(_decref) Task=`b53d6443-efe4-4af1-9d45-50a7c90f6000`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:06:18,462::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:20,584::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:23,463::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:23,464::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1966::DEBUG::2012-08-30 13:06:24,344::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1966::DEBUG::2012-08-30 13:06:24,344::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:06:25,585::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:06:25,847::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-1967::DEBUG::2012-08-30 13:06:26,644::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1967::DEBUG::2012-08-30 13:06:26,645::task::588::TaskManager.Task::(_updateState) Task=`f670d837-a62b-44b6-95bb-6dcfcf337953`::moving from state init -> state preparing
>Thread-1967::INFO::2012-08-30 13:06:26,645::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1967::INFO::2012-08-30 13:06:26,645::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1967::DEBUG::2012-08-30 13:06:26,646::task::1172::TaskManager.Task::(prepare) Task=`f670d837-a62b-44b6-95bb-6dcfcf337953`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1967::DEBUG::2012-08-30 13:06:26,646::task::588::TaskManager.Task::(_updateState) Task=`f670d837-a62b-44b6-95bb-6dcfcf337953`::moving from state preparing -> state finished
>Thread-1967::DEBUG::2012-08-30 13:06:26,646::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1967::DEBUG::2012-08-30 13:06:26,646::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1967::DEBUG::2012-08-30 13:06:26,647::task::978::TaskManager.Task::(_decref) Task=`f670d837-a62b-44b6-95bb-6dcfcf337953`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:06:28,465::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1969::DEBUG::2012-08-30 13:06:29,615::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1969::DEBUG::2012-08-30 13:06:29,616::task::588::TaskManager.Task::(_updateState) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::moving from state init -> state preparing
>Thread-1969::INFO::2012-08-30 13:06:29,616::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1969::DEBUG::2012-08-30 13:06:29,617::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`eb0298bb-b2c8-4d9f-8dfd-65963dc04db1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1969::DEBUG::2012-08-30 13:06:29,617::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1969::DEBUG::2012-08-30 13:06:29,617::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1969::DEBUG::2012-08-30 13:06:29,617::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`eb0298bb-b2c8-4d9f-8dfd-65963dc04db1`::Granted request
>Thread-1969::DEBUG::2012-08-30 13:06:29,618::task::817::TaskManager.Task::(resourceAcquired) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1969::DEBUG::2012-08-30 13:06:29,618::task::978::TaskManager.Task::(_decref) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::ref 1 aborting False
>Thread-1969::INFO::2012-08-30 13:06:29,621::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1969::DEBUG::2012-08-30 13:06:29,621::task::1172::TaskManager.Task::(prepare) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1969::DEBUG::2012-08-30 13:06:29,621::task::588::TaskManager.Task::(_updateState) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::moving from state preparing -> state finished
>Thread-1969::DEBUG::2012-08-30 13:06:29,622::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1969::DEBUG::2012-08-30 13:06:29,622::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1969::DEBUG::2012-08-30 13:06:29,622::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1969::DEBUG::2012-08-30 13:06:29,622::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1969::DEBUG::2012-08-30 13:06:29,623::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1969::DEBUG::2012-08-30 13:06:29,623::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1969::DEBUG::2012-08-30 13:06:29,623::task::978::TaskManager.Task::(_decref) Task=`93ae078a-0a22-4b62-a254-8c373ca28724`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 13:06:30,586::task::588::TaskManager.Task::(_updateState) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:06:30,586::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:06:30,586::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e53ced6e-f5da-488a-b53a-a30c83435706`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:06:30,587::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:06:30,587::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:06:30,587::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e53ced6e-f5da-488a-b53a-a30c83435706`::Granted request
>Thread-104::DEBUG::2012-08-30 13:06:30,587::task::817::TaskManager.Task::(resourceAcquired) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:06:30,588::task::978::TaskManager.Task::(_decref) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:06:30,589::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:06:30,591::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:06:30,592::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:06:30,593::task::1172::TaskManager.Task::(prepare) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:06:30,593::task::588::TaskManager.Task::(_updateState) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:06:30,593::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:06:30,593::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:06:30,594::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:06:30,594::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:06:30,594::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:06:30,594::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:06:30,595::task::978::TaskManager.Task::(_decref) Task=`dfd1c882-d478-4d04-92b8-f0ff3985bcca`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:06:30,595::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:30,595::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:30,596::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:30,596::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:06:32,852::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:06:33,466::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:35,597::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:38,467::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:38,467::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:40,598::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1972::DEBUG::2012-08-30 13:06:42,829::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1972::DEBUG::2012-08-30 13:06:42,830::task::588::TaskManager.Task::(_updateState) Task=`0d295874-7fc1-4786-b90e-43bdbc7e4142`::moving from state init -> state preparing
>Thread-1972::INFO::2012-08-30 13:06:42,830::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1972::INFO::2012-08-30 13:06:42,830::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1972::DEBUG::2012-08-30 13:06:42,830::task::1172::TaskManager.Task::(prepare) Task=`0d295874-7fc1-4786-b90e-43bdbc7e4142`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1972::DEBUG::2012-08-30 13:06:42,831::task::588::TaskManager.Task::(_updateState) Task=`0d295874-7fc1-4786-b90e-43bdbc7e4142`::moving from state preparing -> state finished
>Thread-1972::DEBUG::2012-08-30 13:06:42,831::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1972::DEBUG::2012-08-30 13:06:42,831::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1972::DEBUG::2012-08-30 13:06:42,831::task::978::TaskManager.Task::(_decref) Task=`0d295874-7fc1-4786-b90e-43bdbc7e4142`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:06:43,468::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:45,600::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:45,600::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1974::DEBUG::2012-08-30 13:06:46,149::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1974::DEBUG::2012-08-30 13:06:46,149::task::588::TaskManager.Task::(_updateState) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::moving from state init -> state preparing
>Thread-1974::INFO::2012-08-30 13:06:46,149::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1974::DEBUG::2012-08-30 13:06:46,150::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9d08d300-6c0e-4079-9428-0014b2b000b9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1974::DEBUG::2012-08-30 13:06:46,150::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1974::DEBUG::2012-08-30 13:06:46,150::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1974::DEBUG::2012-08-30 13:06:46,150::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9d08d300-6c0e-4079-9428-0014b2b000b9`::Granted request
>Thread-1974::DEBUG::2012-08-30 13:06:46,151::task::817::TaskManager.Task::(resourceAcquired) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1974::DEBUG::2012-08-30 13:06:46,151::task::978::TaskManager.Task::(_decref) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::ref 1 aborting False
>Thread-1974::INFO::2012-08-30 13:06:46,154::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1974::DEBUG::2012-08-30 13:06:46,154::task::1172::TaskManager.Task::(prepare) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1974::DEBUG::2012-08-30 13:06:46,154::task::588::TaskManager.Task::(_updateState) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::moving from state preparing -> state finished
>Thread-1974::DEBUG::2012-08-30 13:06:46,155::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1974::DEBUG::2012-08-30 13:06:46,155::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1974::DEBUG::2012-08-30 13:06:46,155::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1974::DEBUG::2012-08-30 13:06:46,155::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1974::DEBUG::2012-08-30 13:06:46,156::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1974::DEBUG::2012-08-30 13:06:46,156::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1974::DEBUG::2012-08-30 13:06:46,156::task::978::TaskManager.Task::(_decref) Task=`32d40cb3-1cec-4bc7-ab4f-72dedc2b7855`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:06:48,469::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1975::DEBUG::2012-08-30 13:06:49,813::task::588::TaskManager.Task::(_updateState) Task=`119154e5-5d95-40f2-9cf7-ae5fe307cd09`::moving from state init -> state preparing
>Thread-1975::INFO::2012-08-30 13:06:49,813::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1975::INFO::2012-08-30 13:06:49,814::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161385536194', 'lastCheck': 1346303200.6500571, 'code': 0, 'valid': True}}
>Thread-1975::DEBUG::2012-08-30 13:06:49,814::task::1172::TaskManager.Task::(prepare) Task=`119154e5-5d95-40f2-9cf7-ae5fe307cd09`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161385536194', 'lastCheck': 1346303200.6500571, 'code': 0, 'valid': True}}
>Thread-1975::DEBUG::2012-08-30 13:06:49,814::task::588::TaskManager.Task::(_updateState) Task=`119154e5-5d95-40f2-9cf7-ae5fe307cd09`::moving from state preparing -> state finished
>Thread-1975::DEBUG::2012-08-30 13:06:49,814::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1975::DEBUG::2012-08-30 13:06:49,815::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1975::DEBUG::2012-08-30 13:06:49,815::task::978::TaskManager.Task::(_decref) Task=`119154e5-5d95-40f2-9cf7-ae5fe307cd09`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:06:50,602::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1976::DEBUG::2012-08-30 13:06:52,997::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1976::DEBUG::2012-08-30 13:06:52,998::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::DEBUG::2012-08-30 13:06:53,470::task::588::TaskManager.Task::(_updateState) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:06:53,471::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:06:53,471::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`526dac2c-5454-4a19-866f-20d66f2f78eb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:06:53,471::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:06:53,471::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:06:53,472::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`526dac2c-5454-4a19-866f-20d66f2f78eb`::Granted request
>Thread-40::DEBUG::2012-08-30 13:06:53,472::task::817::TaskManager.Task::(resourceAcquired) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:06:53,472::task::978::TaskManager.Task::(_decref) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:06:53,474::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:06:53,476::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:06:53,477::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:06:53,477::task::1172::TaskManager.Task::(prepare) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:06:53,478::task::588::TaskManager.Task::(_updateState) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:06:53,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:06:53,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:06:53,478::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:06:53,479::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:06:53,479::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:06:53,479::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:06:53,479::task::978::TaskManager.Task::(_decref) Task=`e04cba6b-a0ed-4c09-86b7-b80c38a06ed6`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:06:53,480::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:53,480::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:53,481::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:06:53,481::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:06:55,603::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:06:55,871::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:06:58,482::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1978::DEBUG::2012-08-30 13:06:59,352::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1978::DEBUG::2012-08-30 13:06:59,353::task::588::TaskManager.Task::(_updateState) Task=`a5c391d1-a768-445b-970e-395abf33b8f2`::moving from state init -> state preparing
>Thread-1978::INFO::2012-08-30 13:06:59,353::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1978::INFO::2012-08-30 13:06:59,353::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1978::DEBUG::2012-08-30 13:06:59,353::task::1172::TaskManager.Task::(prepare) Task=`a5c391d1-a768-445b-970e-395abf33b8f2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1978::DEBUG::2012-08-30 13:06:59,354::task::588::TaskManager.Task::(_updateState) Task=`a5c391d1-a768-445b-970e-395abf33b8f2`::moving from state preparing -> state finished
>Thread-1978::DEBUG::2012-08-30 13:06:59,354::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1978::DEBUG::2012-08-30 13:06:59,354::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1978::DEBUG::2012-08-30 13:06:59,354::task::978::TaskManager.Task::(_decref) Task=`a5c391d1-a768-445b-970e-395abf33b8f2`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:07:00,604::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:00,605::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1979::DEBUG::2012-08-30 13:07:02,489::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1979::DEBUG::2012-08-30 13:07:02,490::task::588::TaskManager.Task::(_updateState) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::moving from state init -> state preparing
>Thread-1979::INFO::2012-08-30 13:07:02,490::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1979::DEBUG::2012-08-30 13:07:02,490::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9ebdc2cd-5a5f-4acc-a690-4031dd19d85c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1979::DEBUG::2012-08-30 13:07:02,490::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1979::DEBUG::2012-08-30 13:07:02,491::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1979::DEBUG::2012-08-30 13:07:02,491::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9ebdc2cd-5a5f-4acc-a690-4031dd19d85c`::Granted request
>Thread-1979::DEBUG::2012-08-30 13:07:02,491::task::817::TaskManager.Task::(resourceAcquired) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1979::DEBUG::2012-08-30 13:07:02,492::task::978::TaskManager.Task::(_decref) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::ref 1 aborting False
>Thread-1979::INFO::2012-08-30 13:07:02,495::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1979::DEBUG::2012-08-30 13:07:02,495::task::1172::TaskManager.Task::(prepare) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1979::DEBUG::2012-08-30 13:07:02,495::task::588::TaskManager.Task::(_updateState) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::moving from state preparing -> state finished
>Thread-1979::DEBUG::2012-08-30 13:07:02,495::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1979::DEBUG::2012-08-30 13:07:02,495::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1979::DEBUG::2012-08-30 13:07:02,496::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1979::DEBUG::2012-08-30 13:07:02,496::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1979::DEBUG::2012-08-30 13:07:02,496::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1979::DEBUG::2012-08-30 13:07:02,496::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1979::DEBUG::2012-08-30 13:07:02,497::task::978::TaskManager.Task::(_decref) Task=`7c449eb0-72b1-4beb-b815-a16b2f8b4414`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:07:02,878::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:07:03,483::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:05,606::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:08,484::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:08,484::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:10,607::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:13,485::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:15,608::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:15,609::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1983::DEBUG::2012-08-30 13:07:15,853::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1983::DEBUG::2012-08-30 13:07:15,853::task::588::TaskManager.Task::(_updateState) Task=`2becd3dc-016e-465f-abaf-5dee715d6856`::moving from state init -> state preparing
>Thread-1983::INFO::2012-08-30 13:07:15,853::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1983::INFO::2012-08-30 13:07:15,854::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1983::DEBUG::2012-08-30 13:07:15,854::task::1172::TaskManager.Task::(prepare) Task=`2becd3dc-016e-465f-abaf-5dee715d6856`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1983::DEBUG::2012-08-30 13:07:15,854::task::588::TaskManager.Task::(_updateState) Task=`2becd3dc-016e-465f-abaf-5dee715d6856`::moving from state preparing -> state finished
>Thread-1983::DEBUG::2012-08-30 13:07:15,854::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1983::DEBUG::2012-08-30 13:07:15,855::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1983::DEBUG::2012-08-30 13:07:15,855::task::978::TaskManager.Task::(_decref) Task=`2becd3dc-016e-465f-abaf-5dee715d6856`::ref 0 aborting False
>Thread-1984::DEBUG::2012-08-30 13:07:18,130::task::588::TaskManager.Task::(_updateState) Task=`1d048f47-6225-413e-8590-e1d890f796a5`::moving from state init -> state preparing
>Thread-1984::INFO::2012-08-30 13:07:18,131::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1984::INFO::2012-08-30 13:07:18,131::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163507461548', 'lastCheck': 1346303230.670476, 'code': 0, 'valid': True}}
>Thread-1984::DEBUG::2012-08-30 13:07:18,131::task::1172::TaskManager.Task::(prepare) Task=`1d048f47-6225-413e-8590-e1d890f796a5`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163507461548', 'lastCheck': 1346303230.670476, 'code': 0, 'valid': True}}
>Thread-1984::DEBUG::2012-08-30 13:07:18,131::task::588::TaskManager.Task::(_updateState) Task=`1d048f47-6225-413e-8590-e1d890f796a5`::moving from state preparing -> state finished
>Thread-1984::DEBUG::2012-08-30 13:07:18,132::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1984::DEBUG::2012-08-30 13:07:18,132::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1984::DEBUG::2012-08-30 13:07:18,132::task::978::TaskManager.Task::(_decref) Task=`1d048f47-6225-413e-8590-e1d890f796a5`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:07:18,486::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1985::DEBUG::2012-08-30 13:07:19,007::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1985::DEBUG::2012-08-30 13:07:19,008::task::588::TaskManager.Task::(_updateState) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::moving from state init -> state preparing
>Thread-1985::INFO::2012-08-30 13:07:19,008::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1985::DEBUG::2012-08-30 13:07:19,008::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d5d377b3-0690-44fd-9e52-5a8c3b905d92`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1985::DEBUG::2012-08-30 13:07:19,008::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1985::DEBUG::2012-08-30 13:07:19,009::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1985::DEBUG::2012-08-30 13:07:19,009::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d5d377b3-0690-44fd-9e52-5a8c3b905d92`::Granted request
>Thread-1985::DEBUG::2012-08-30 13:07:19,009::task::817::TaskManager.Task::(resourceAcquired) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1985::DEBUG::2012-08-30 13:07:19,009::task::978::TaskManager.Task::(_decref) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::ref 1 aborting False
>Thread-1985::INFO::2012-08-30 13:07:19,012::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1985::DEBUG::2012-08-30 13:07:19,013::task::1172::TaskManager.Task::(prepare) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1985::DEBUG::2012-08-30 13:07:19,013::task::588::TaskManager.Task::(_updateState) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::moving from state preparing -> state finished
>Thread-1985::DEBUG::2012-08-30 13:07:19,013::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1985::DEBUG::2012-08-30 13:07:19,013::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1985::DEBUG::2012-08-30 13:07:19,014::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1985::DEBUG::2012-08-30 13:07:19,014::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1985::DEBUG::2012-08-30 13:07:19,014::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1985::DEBUG::2012-08-30 13:07:19,014::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1985::DEBUG::2012-08-30 13:07:19,015::task::978::TaskManager.Task::(_decref) Task=`7312dd3e-ebc0-4162-b942-f68d5c9ffc15`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:07:20,610::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1986::DEBUG::2012-08-30 13:07:21,294::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1986::DEBUG::2012-08-30 13:07:21,295::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:07:23,487::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:23,488::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:25,611::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:07:25,894::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:07:28,489::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:07:30,612::task::588::TaskManager.Task::(_updateState) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:07:30,613::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:07:30,613::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d504b74a-495e-4caf-acc0-8e8377277cf9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:07:30,613::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:07:30,614::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:07:30,614::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d504b74a-495e-4caf-acc0-8e8377277cf9`::Granted request
>Thread-104::DEBUG::2012-08-30 13:07:30,614::task::817::TaskManager.Task::(resourceAcquired) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:07:30,615::task::978::TaskManager.Task::(_decref) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:07:30,616::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:07:30,618::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:07:30,619::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:07:30,620::task::1172::TaskManager.Task::(prepare) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:07:30,620::task::588::TaskManager.Task::(_updateState) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:07:30,620::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:07:30,620::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:07:30,620::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:07:30,621::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:07:30,621::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:07:30,621::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:07:30,621::task::978::TaskManager.Task::(_decref) Task=`8bce6b29-10b3-4be8-8298-32c43937613d`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:07:30,622::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:30,622::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:30,623::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:30,623::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1989::DEBUG::2012-08-30 13:07:31,724::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1989::DEBUG::2012-08-30 13:07:31,725::task::588::TaskManager.Task::(_updateState) Task=`7e5cb8ad-88cd-4571-b6e8-5079c081cc09`::moving from state init -> state preparing
>Thread-1989::INFO::2012-08-30 13:07:31,725::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1989::INFO::2012-08-30 13:07:31,726::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1989::DEBUG::2012-08-30 13:07:31,726::task::1172::TaskManager.Task::(prepare) Task=`7e5cb8ad-88cd-4571-b6e8-5079c081cc09`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1989::DEBUG::2012-08-30 13:07:31,726::task::588::TaskManager.Task::(_updateState) Task=`7e5cb8ad-88cd-4571-b6e8-5079c081cc09`::moving from state preparing -> state finished
>Thread-1989::DEBUG::2012-08-30 13:07:31,726::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1989::DEBUG::2012-08-30 13:07:31,726::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1989::DEBUG::2012-08-30 13:07:31,727::task::978::TaskManager.Task::(_decref) Task=`7e5cb8ad-88cd-4571-b6e8-5079c081cc09`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:07:32,900::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:07:33,490::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1990::DEBUG::2012-08-30 13:07:34,891::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1990::DEBUG::2012-08-30 13:07:34,891::task::588::TaskManager.Task::(_updateState) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::moving from state init -> state preparing
>Thread-1990::INFO::2012-08-30 13:07:34,891::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1990::DEBUG::2012-08-30 13:07:34,892::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`08b65a24-6d85-4624-a92b-95f45bf21f06`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1990::DEBUG::2012-08-30 13:07:34,892::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1990::DEBUG::2012-08-30 13:07:34,892::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1990::DEBUG::2012-08-30 13:07:34,893::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`08b65a24-6d85-4624-a92b-95f45bf21f06`::Granted request
>Thread-1990::DEBUG::2012-08-30 13:07:34,893::task::817::TaskManager.Task::(resourceAcquired) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1990::DEBUG::2012-08-30 13:07:34,893::task::978::TaskManager.Task::(_decref) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::ref 1 aborting False
>Thread-1990::INFO::2012-08-30 13:07:34,896::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1990::DEBUG::2012-08-30 13:07:34,896::task::1172::TaskManager.Task::(prepare) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1990::DEBUG::2012-08-30 13:07:34,897::task::588::TaskManager.Task::(_updateState) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::moving from state preparing -> state finished
>Thread-1990::DEBUG::2012-08-30 13:07:34,897::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1990::DEBUG::2012-08-30 13:07:34,897::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1990::DEBUG::2012-08-30 13:07:34,897::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1990::DEBUG::2012-08-30 13:07:34,898::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1990::DEBUG::2012-08-30 13:07:34,898::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1990::DEBUG::2012-08-30 13:07:34,898::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1990::DEBUG::2012-08-30 13:07:34,898::task::978::TaskManager.Task::(_decref) Task=`17a690da-5582-4b7f-90fa-eede1801bcb9`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:07:35,624::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:38,491::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:38,491::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:40,625::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:43,492::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:45,627::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:45,627::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1993::DEBUG::2012-08-30 13:07:46,559::task::588::TaskManager.Task::(_updateState) Task=`d278eb17-f298-429a-abe2-05adc1b7c030`::moving from state init -> state preparing
>Thread-1993::INFO::2012-08-30 13:07:46,559::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-1993::INFO::2012-08-30 13:07:46,560::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346303260.6891639, 'code': 0, 'valid': True}}
>Thread-1993::DEBUG::2012-08-30 13:07:46,560::task::1172::TaskManager.Task::(prepare) Task=`d278eb17-f298-429a-abe2-05adc1b7c030`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346303260.6891639, 'code': 0, 'valid': True}}
>Thread-1993::DEBUG::2012-08-30 13:07:46,560::task::588::TaskManager.Task::(_updateState) Task=`d278eb17-f298-429a-abe2-05adc1b7c030`::moving from state preparing -> state finished
>Thread-1993::DEBUG::2012-08-30 13:07:46,560::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1993::DEBUG::2012-08-30 13:07:46,561::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1993::DEBUG::2012-08-30 13:07:46,561::task::978::TaskManager.Task::(_decref) Task=`d278eb17-f298-429a-abe2-05adc1b7c030`::ref 0 aborting False
>Thread-1994::DEBUG::2012-08-30 13:07:48,138::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1994::DEBUG::2012-08-30 13:07:48,139::task::588::TaskManager.Task::(_updateState) Task=`1ee5ed9e-3d77-4a4c-bb13-8f47ead90d7d`::moving from state init -> state preparing
>Thread-1994::INFO::2012-08-30 13:07:48,139::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1994::INFO::2012-08-30 13:07:48,139::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1994::DEBUG::2012-08-30 13:07:48,140::task::1172::TaskManager.Task::(prepare) Task=`1ee5ed9e-3d77-4a4c-bb13-8f47ead90d7d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1994::DEBUG::2012-08-30 13:07:48,140::task::588::TaskManager.Task::(_updateState) Task=`1ee5ed9e-3d77-4a4c-bb13-8f47ead90d7d`::moving from state preparing -> state finished
>Thread-1994::DEBUG::2012-08-30 13:07:48,140::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1994::DEBUG::2012-08-30 13:07:48,140::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1994::DEBUG::2012-08-30 13:07:48,140::task::978::TaskManager.Task::(_decref) Task=`1ee5ed9e-3d77-4a4c-bb13-8f47ead90d7d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:07:48,493::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1995::DEBUG::2012-08-30 13:07:49,715::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-1995::DEBUG::2012-08-30 13:07:49,716::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:07:50,628::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1996::DEBUG::2012-08-30 13:07:51,299::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1996::DEBUG::2012-08-30 13:07:51,299::task::588::TaskManager.Task::(_updateState) Task=`75900655-3885-48a7-9afd-379a5fd98766`::moving from state init -> state preparing
>Thread-1996::INFO::2012-08-30 13:07:51,300::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1996::DEBUG::2012-08-30 13:07:51,300::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`96b96da8-9b7a-4744-bf98-69a6e041329e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-1996::DEBUG::2012-08-30 13:07:51,300::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-1996::DEBUG::2012-08-30 13:07:51,301::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-1996::DEBUG::2012-08-30 13:07:51,301::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`96b96da8-9b7a-4744-bf98-69a6e041329e`::Granted request
>Thread-1996::DEBUG::2012-08-30 13:07:51,301::task::817::TaskManager.Task::(resourceAcquired) Task=`75900655-3885-48a7-9afd-379a5fd98766`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-1996::DEBUG::2012-08-30 13:07:51,301::task::978::TaskManager.Task::(_decref) Task=`75900655-3885-48a7-9afd-379a5fd98766`::ref 1 aborting False
>Thread-1996::INFO::2012-08-30 13:07:51,304::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1996::DEBUG::2012-08-30 13:07:51,305::task::1172::TaskManager.Task::(prepare) Task=`75900655-3885-48a7-9afd-379a5fd98766`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-1996::DEBUG::2012-08-30 13:07:51,305::task::588::TaskManager.Task::(_updateState) Task=`75900655-3885-48a7-9afd-379a5fd98766`::moving from state preparing -> state finished
>Thread-1996::DEBUG::2012-08-30 13:07:51,305::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-1996::DEBUG::2012-08-30 13:07:51,305::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1996::DEBUG::2012-08-30 13:07:51,306::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-1996::DEBUG::2012-08-30 13:07:51,306::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-1996::DEBUG::2012-08-30 13:07:51,306::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-1996::DEBUG::2012-08-30 13:07:51,306::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-1996::DEBUG::2012-08-30 13:07:51,306::task::978::TaskManager.Task::(_decref) Task=`75900655-3885-48a7-9afd-379a5fd98766`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 13:07:53,494::task::588::TaskManager.Task::(_updateState) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:07:53,494::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:07:53,495::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9ecbe519-7f13-4a11-aa56-6abc51dbc34c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:07:53,495::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:07:53,495::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:07:53,495::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9ecbe519-7f13-4a11-aa56-6abc51dbc34c`::Granted request
>Thread-40::DEBUG::2012-08-30 13:07:53,496::task::817::TaskManager.Task::(resourceAcquired) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:07:53,496::task::978::TaskManager.Task::(_decref) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:07:53,498::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:07:53,500::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:07:53,501::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:07:53,501::task::1172::TaskManager.Task::(prepare) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:07:53,501::task::588::TaskManager.Task::(_updateState) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:07:53,502::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:07:53,502::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:07:53,502::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:07:53,502::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:07:53,503::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:07:53,503::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:07:53,503::task::978::TaskManager.Task::(_decref) Task=`47f6bd23-04b3-4a85-97b0-4ad4165a3dc2`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:07:53,503::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:53,504::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:53,504::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:07:53,505::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:07:55,630::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:07:55,916::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:07:58,506::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:00,631::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:00,631::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:08:02,923::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:08:03,506::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-1999::DEBUG::2012-08-30 13:08:04,496::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-1999::DEBUG::2012-08-30 13:08:04,496::task::588::TaskManager.Task::(_updateState) Task=`2496c360-e315-4c9b-bbb3-2808fc314f1e`::moving from state init -> state preparing
>Thread-1999::INFO::2012-08-30 13:08:04,496::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-1999::INFO::2012-08-30 13:08:04,497::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1999::DEBUG::2012-08-30 13:08:04,497::task::1172::TaskManager.Task::(prepare) Task=`2496c360-e315-4c9b-bbb3-2808fc314f1e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-1999::DEBUG::2012-08-30 13:08:04,497::task::588::TaskManager.Task::(_updateState) Task=`2496c360-e315-4c9b-bbb3-2808fc314f1e`::moving from state preparing -> state finished
>Thread-1999::DEBUG::2012-08-30 13:08:04,497::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-1999::DEBUG::2012-08-30 13:08:04,498::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-1999::DEBUG::2012-08-30 13:08:04,498::task::978::TaskManager.Task::(_decref) Task=`2496c360-e315-4c9b-bbb3-2808fc314f1e`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:08:05,632::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2001::DEBUG::2012-08-30 13:08:07,762::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2001::DEBUG::2012-08-30 13:08:07,763::task::588::TaskManager.Task::(_updateState) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::moving from state init -> state preparing
>Thread-2001::INFO::2012-08-30 13:08:07,763::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2001::DEBUG::2012-08-30 13:08:07,763::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`576931c6-fb6b-4341-90d2-f531ffa587b2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2001::DEBUG::2012-08-30 13:08:07,764::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2001::DEBUG::2012-08-30 13:08:07,764::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2001::DEBUG::2012-08-30 13:08:07,764::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`576931c6-fb6b-4341-90d2-f531ffa587b2`::Granted request
>Thread-2001::DEBUG::2012-08-30 13:08:07,765::task::817::TaskManager.Task::(resourceAcquired) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2001::DEBUG::2012-08-30 13:08:07,765::task::978::TaskManager.Task::(_decref) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::ref 1 aborting False
>Thread-2001::INFO::2012-08-30 13:08:07,768::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2001::DEBUG::2012-08-30 13:08:07,768::task::1172::TaskManager.Task::(prepare) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2001::DEBUG::2012-08-30 13:08:07,768::task::588::TaskManager.Task::(_updateState) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::moving from state preparing -> state finished
>Thread-2001::DEBUG::2012-08-30 13:08:07,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2001::DEBUG::2012-08-30 13:08:07,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2001::DEBUG::2012-08-30 13:08:07,769::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2001::DEBUG::2012-08-30 13:08:07,769::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2001::DEBUG::2012-08-30 13:08:07,769::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2001::DEBUG::2012-08-30 13:08:07,770::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2001::DEBUG::2012-08-30 13:08:07,770::task::978::TaskManager.Task::(_decref) Task=`73381991-b67d-45f8-9a17-b34967bee0cd`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:08:08,507::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:08,508::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:10,634::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:13,509::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2003::DEBUG::2012-08-30 13:08:15,581::task::588::TaskManager.Task::(_updateState) Task=`e5d3fce7-ee11-4192-8b8f-8796a06411b1`::moving from state init -> state preparing
>Thread-2003::INFO::2012-08-30 13:08:15,581::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2003::INFO::2012-08-30 13:08:15,582::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346303290.70964, 'code': 0, 'valid': True}}
>Thread-2003::DEBUG::2012-08-30 13:08:15,582::task::1172::TaskManager.Task::(prepare) Task=`e5d3fce7-ee11-4192-8b8f-8796a06411b1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346303290.70964, 'code': 0, 'valid': True}}
>Thread-2003::DEBUG::2012-08-30 13:08:15,582::task::588::TaskManager.Task::(_updateState) Task=`e5d3fce7-ee11-4192-8b8f-8796a06411b1`::moving from state preparing -> state finished
>Thread-2003::DEBUG::2012-08-30 13:08:15,582::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2003::DEBUG::2012-08-30 13:08:15,582::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2003::DEBUG::2012-08-30 13:08:15,583::task::978::TaskManager.Task::(_decref) Task=`e5d3fce7-ee11-4192-8b8f-8796a06411b1`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:08:15,635::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:15,636::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:18,510::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2004::DEBUG::2012-08-30 13:08:18,784::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2004::DEBUG::2012-08-30 13:08:18,784::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-2005::DEBUG::2012-08-30 13:08:20,354::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2005::DEBUG::2012-08-30 13:08:20,355::task::588::TaskManager.Task::(_updateState) Task=`f017c5d8-1305-4894-b56a-43dfd9dbede2`::moving from state init -> state preparing
>Thread-2005::INFO::2012-08-30 13:08:20,355::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2005::INFO::2012-08-30 13:08:20,355::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2005::DEBUG::2012-08-30 13:08:20,355::task::1172::TaskManager.Task::(prepare) Task=`f017c5d8-1305-4894-b56a-43dfd9dbede2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2005::DEBUG::2012-08-30 13:08:20,356::task::588::TaskManager.Task::(_updateState) Task=`f017c5d8-1305-4894-b56a-43dfd9dbede2`::moving from state preparing -> state finished
>Thread-2005::DEBUG::2012-08-30 13:08:20,356::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2005::DEBUG::2012-08-30 13:08:20,356::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2005::DEBUG::2012-08-30 13:08:20,356::task::978::TaskManager.Task::(_decref) Task=`f017c5d8-1305-4894-b56a-43dfd9dbede2`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:08:20,637::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:23,511::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:23,511::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2006::DEBUG::2012-08-30 13:08:23,515::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2006::DEBUG::2012-08-30 13:08:23,516::task::588::TaskManager.Task::(_updateState) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::moving from state init -> state preparing
>Thread-2006::INFO::2012-08-30 13:08:23,516::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2006::DEBUG::2012-08-30 13:08:23,516::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ea018b02-bb74-423a-9cba-6fa275236f53`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2006::DEBUG::2012-08-30 13:08:23,517::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2006::DEBUG::2012-08-30 13:08:23,517::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2006::DEBUG::2012-08-30 13:08:23,518::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ea018b02-bb74-423a-9cba-6fa275236f53`::Granted request
>Thread-2006::DEBUG::2012-08-30 13:08:23,518::task::817::TaskManager.Task::(resourceAcquired) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2006::DEBUG::2012-08-30 13:08:23,518::task::978::TaskManager.Task::(_decref) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::ref 1 aborting False
>Thread-2006::INFO::2012-08-30 13:08:23,521::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2006::DEBUG::2012-08-30 13:08:23,521::task::1172::TaskManager.Task::(prepare) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2006::DEBUG::2012-08-30 13:08:23,522::task::588::TaskManager.Task::(_updateState) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::moving from state preparing -> state finished
>Thread-2006::DEBUG::2012-08-30 13:08:23,522::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2006::DEBUG::2012-08-30 13:08:23,522::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2006::DEBUG::2012-08-30 13:08:23,522::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2006::DEBUG::2012-08-30 13:08:23,523::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2006::DEBUG::2012-08-30 13:08:23,523::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2006::DEBUG::2012-08-30 13:08:23,523::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2006::DEBUG::2012-08-30 13:08:23,523::task::978::TaskManager.Task::(_decref) Task=`d7c489e0-c1f6-477f-819a-594fbdc73f6c`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:08:25,638::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:08:25,943::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:08:28,512::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:08:30,639::task::588::TaskManager.Task::(_updateState) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:08:30,640::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:08:30,640::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f08f061c-8b62-403e-a48e-8edbbc716fe5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:08:30,640::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:08:30,640::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:08:30,641::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f08f061c-8b62-403e-a48e-8edbbc716fe5`::Granted request
>Thread-104::DEBUG::2012-08-30 13:08:30,641::task::817::TaskManager.Task::(resourceAcquired) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:08:30,641::task::978::TaskManager.Task::(_decref) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:08:30,643::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:08:30,645::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:08:30,646::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:08:30,646::task::1172::TaskManager.Task::(prepare) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:08:30,646::task::588::TaskManager.Task::(_updateState) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:08:30,647::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:08:30,647::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:08:30,647::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:08:30,647::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:08:30,648::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:08:30,648::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:08:30,648::task::978::TaskManager.Task::(_decref) Task=`92235e6a-e6fe-4621-acb3-9597227a046a`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:08:30,649::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:30,649::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:30,649::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:30,650::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:08:32,947::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:08:33,513::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:35,651::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2010::DEBUG::2012-08-30 13:08:36,698::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2010::DEBUG::2012-08-30 13:08:36,699::task::588::TaskManager.Task::(_updateState) Task=`679a1e33-2ad5-4e72-bbd0-1349e6e4dbc6`::moving from state init -> state preparing
>Thread-2010::INFO::2012-08-30 13:08:36,699::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2010::INFO::2012-08-30 13:08:36,699::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2010::DEBUG::2012-08-30 13:08:36,699::task::1172::TaskManager.Task::(prepare) Task=`679a1e33-2ad5-4e72-bbd0-1349e6e4dbc6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2010::DEBUG::2012-08-30 13:08:36,700::task::588::TaskManager.Task::(_updateState) Task=`679a1e33-2ad5-4e72-bbd0-1349e6e4dbc6`::moving from state preparing -> state finished
>Thread-2010::DEBUG::2012-08-30 13:08:36,700::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2010::DEBUG::2012-08-30 13:08:36,700::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2010::DEBUG::2012-08-30 13:08:36,700::task::978::TaskManager.Task::(_decref) Task=`679a1e33-2ad5-4e72-bbd0-1349e6e4dbc6`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:08:38,514::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:38,515::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2012::DEBUG::2012-08-30 13:08:39,842::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2012::DEBUG::2012-08-30 13:08:39,843::task::588::TaskManager.Task::(_updateState) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::moving from state init -> state preparing
>Thread-2012::INFO::2012-08-30 13:08:39,843::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2012::DEBUG::2012-08-30 13:08:39,843::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`362c63b4-fbca-4040-b737-e7432b5bf585`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2012::DEBUG::2012-08-30 13:08:39,843::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2012::DEBUG::2012-08-30 13:08:39,844::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2012::DEBUG::2012-08-30 13:08:39,844::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`362c63b4-fbca-4040-b737-e7432b5bf585`::Granted request
>Thread-2012::DEBUG::2012-08-30 13:08:39,844::task::817::TaskManager.Task::(resourceAcquired) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2012::DEBUG::2012-08-30 13:08:39,844::task::978::TaskManager.Task::(_decref) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::ref 1 aborting False
>Thread-2012::INFO::2012-08-30 13:08:39,847::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2012::DEBUG::2012-08-30 13:08:39,848::task::1172::TaskManager.Task::(prepare) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2012::DEBUG::2012-08-30 13:08:39,848::task::588::TaskManager.Task::(_updateState) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::moving from state preparing -> state finished
>Thread-2012::DEBUG::2012-08-30 13:08:39,848::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2012::DEBUG::2012-08-30 13:08:39,848::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2012::DEBUG::2012-08-30 13:08:39,849::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2012::DEBUG::2012-08-30 13:08:39,849::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2012::DEBUG::2012-08-30 13:08:39,849::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2012::DEBUG::2012-08-30 13:08:39,849::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2012::DEBUG::2012-08-30 13:08:39,850::task::978::TaskManager.Task::(_decref) Task=`8bc03bff-d459-4284-a897-396d8b5afaa5`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:08:40,652::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:43,515::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2013::DEBUG::2012-08-30 13:08:44,154::task::588::TaskManager.Task::(_updateState) Task=`291e2646-a082-4b65-b3ac-528435a71888`::moving from state init -> state preparing
>Thread-2013::INFO::2012-08-30 13:08:44,154::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2013::INFO::2012-08-30 13:08:44,155::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163006782532', 'lastCheck': 1346303320.728425, 'code': 0, 'valid': True}}
>Thread-2013::DEBUG::2012-08-30 13:08:44,155::task::1172::TaskManager.Task::(prepare) Task=`291e2646-a082-4b65-b3ac-528435a71888`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163006782532', 'lastCheck': 1346303320.728425, 'code': 0, 'valid': True}}
>Thread-2013::DEBUG::2012-08-30 13:08:44,155::task::588::TaskManager.Task::(_updateState) Task=`291e2646-a082-4b65-b3ac-528435a71888`::moving from state preparing -> state finished
>Thread-2013::DEBUG::2012-08-30 13:08:44,155::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2013::DEBUG::2012-08-30 13:08:44,156::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2013::DEBUG::2012-08-30 13:08:44,156::task::978::TaskManager.Task::(_decref) Task=`291e2646-a082-4b65-b3ac-528435a71888`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:08:45,654::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:45,654::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2014::DEBUG::2012-08-30 13:08:47,335::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2014::DEBUG::2012-08-30 13:08:47,336::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:08:48,516::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:50,655::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2016::DEBUG::2012-08-30 13:08:52,934::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2016::DEBUG::2012-08-30 13:08:52,934::task::588::TaskManager.Task::(_updateState) Task=`5fa5fe1f-df7b-419d-a520-3234c010315c`::moving from state init -> state preparing
>Thread-2016::INFO::2012-08-30 13:08:52,935::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2016::INFO::2012-08-30 13:08:52,935::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2016::DEBUG::2012-08-30 13:08:52,935::task::1172::TaskManager.Task::(prepare) Task=`5fa5fe1f-df7b-419d-a520-3234c010315c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2016::DEBUG::2012-08-30 13:08:52,936::task::588::TaskManager.Task::(_updateState) Task=`5fa5fe1f-df7b-419d-a520-3234c010315c`::moving from state preparing -> state finished
>Thread-2016::DEBUG::2012-08-30 13:08:52,936::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2016::DEBUG::2012-08-30 13:08:52,936::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2016::DEBUG::2012-08-30 13:08:52,936::task::978::TaskManager.Task::(_decref) Task=`5fa5fe1f-df7b-419d-a520-3234c010315c`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 13:08:53,517::task::588::TaskManager.Task::(_updateState) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:08:53,518::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:08:53,518::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`b13fd91b-e2d2-4422-a0c0-d0908adfddb6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:08:53,518::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:08:53,518::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:08:53,519::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`b13fd91b-e2d2-4422-a0c0-d0908adfddb6`::Granted request
>Thread-40::DEBUG::2012-08-30 13:08:53,519::task::817::TaskManager.Task::(resourceAcquired) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:08:53,519::task::978::TaskManager.Task::(_decref) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:08:53,521::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:08:53,523::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:08:53,524::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:08:53,524::task::1172::TaskManager.Task::(prepare) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:08:53,525::task::588::TaskManager.Task::(_updateState) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:08:53,525::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:08:53,525::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:08:53,525::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:08:53,526::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:08:53,526::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:08:53,526::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:08:53,526::task::978::TaskManager.Task::(_decref) Task=`1181f604-7e5c-4934-908b-37c49e19aaf6`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:08:53,527::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:53,527::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:53,528::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:08:53,528::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:08:55,657::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:08:55,966::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2017::DEBUG::2012-08-30 13:08:56,084::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2017::DEBUG::2012-08-30 13:08:56,085::task::588::TaskManager.Task::(_updateState) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::moving from state init -> state preparing
>Thread-2017::INFO::2012-08-30 13:08:56,085::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2017::DEBUG::2012-08-30 13:08:56,085::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`975c8ec8-c6ce-410d-a9ca-418af24c6484`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2017::DEBUG::2012-08-30 13:08:56,086::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2017::DEBUG::2012-08-30 13:08:56,086::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2017::DEBUG::2012-08-30 13:08:56,086::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`975c8ec8-c6ce-410d-a9ca-418af24c6484`::Granted request
>Thread-2017::DEBUG::2012-08-30 13:08:56,086::task::817::TaskManager.Task::(resourceAcquired) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2017::DEBUG::2012-08-30 13:08:56,087::task::978::TaskManager.Task::(_decref) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::ref 1 aborting False
>Thread-2017::INFO::2012-08-30 13:08:56,090::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2017::DEBUG::2012-08-30 13:08:56,090::task::1172::TaskManager.Task::(prepare) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2017::DEBUG::2012-08-30 13:08:56,090::task::588::TaskManager.Task::(_updateState) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::moving from state preparing -> state finished
>Thread-2017::DEBUG::2012-08-30 13:08:56,090::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2017::DEBUG::2012-08-30 13:08:56,091::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2017::DEBUG::2012-08-30 13:08:56,091::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2017::DEBUG::2012-08-30 13:08:56,091::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2017::DEBUG::2012-08-30 13:08:56,091::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2017::DEBUG::2012-08-30 13:08:56,092::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2017::DEBUG::2012-08-30 13:08:56,092::task::978::TaskManager.Task::(_decref) Task=`eee8f1a8-0b47-4c5a-a496-e5148b244dbe`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:08:58,529::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:00,658::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:00,658::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:09:02,973::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:09:03,530::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:05,659::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:08,531::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:08,531::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2021::DEBUG::2012-08-30 13:09:09,477::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2021::DEBUG::2012-08-30 13:09:09,477::task::588::TaskManager.Task::(_updateState) Task=`42319c1e-eb08-479a-8ee9-57703ffcd3b5`::moving from state init -> state preparing
>Thread-2021::INFO::2012-08-30 13:09:09,477::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2021::INFO::2012-08-30 13:09:09,478::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2021::DEBUG::2012-08-30 13:09:09,478::task::1172::TaskManager.Task::(prepare) Task=`42319c1e-eb08-479a-8ee9-57703ffcd3b5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2021::DEBUG::2012-08-30 13:09:09,478::task::588::TaskManager.Task::(_updateState) Task=`42319c1e-eb08-479a-8ee9-57703ffcd3b5`::moving from state preparing -> state finished
>Thread-2021::DEBUG::2012-08-30 13:09:09,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2021::DEBUG::2012-08-30 13:09:09,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2021::DEBUG::2012-08-30 13:09:09,479::task::978::TaskManager.Task::(_decref) Task=`42319c1e-eb08-479a-8ee9-57703ffcd3b5`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:09:10,660::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2022::DEBUG::2012-08-30 13:09:12,895::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2023::DEBUG::2012-08-30 13:09:12,896::task::588::TaskManager.Task::(_updateState) Task=`73d3a043-0fe9-4160-9539-44ffced6508f`::moving from state init -> state preparing
>Thread-2022::DEBUG::2012-08-30 13:09:12,896::task::588::TaskManager.Task::(_updateState) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::moving from state init -> state preparing
>Thread-2023::INFO::2012-08-30 13:09:12,896::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2022::INFO::2012-08-30 13:09:12,896::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2023::INFO::2012-08-30 13:09:12,897::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164389610291', 'lastCheck': 1346303350.7488611, 'code': 0, 'valid': True}}
>Thread-2022::DEBUG::2012-08-30 13:09:12,897::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`06036aad-493a-4d68-9987-2efbdbb4c731`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2023::DEBUG::2012-08-30 13:09:12,897::task::1172::TaskManager.Task::(prepare) Task=`73d3a043-0fe9-4160-9539-44ffced6508f`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164389610291', 'lastCheck': 1346303350.7488611, 'code': 0, 'valid': True}}
>Thread-2022::DEBUG::2012-08-30 13:09:12,897::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2023::DEBUG::2012-08-30 13:09:12,898::task::588::TaskManager.Task::(_updateState) Task=`73d3a043-0fe9-4160-9539-44ffced6508f`::moving from state preparing -> state finished
>Thread-2022::DEBUG::2012-08-30 13:09:12,898::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2023::DEBUG::2012-08-30 13:09:12,898::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2022::DEBUG::2012-08-30 13:09:12,898::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`06036aad-493a-4d68-9987-2efbdbb4c731`::Granted request
>Thread-2023::DEBUG::2012-08-30 13:09:12,898::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2023::DEBUG::2012-08-30 13:09:12,899::task::978::TaskManager.Task::(_decref) Task=`73d3a043-0fe9-4160-9539-44ffced6508f`::ref 0 aborting False
>Thread-2022::DEBUG::2012-08-30 13:09:12,900::task::817::TaskManager.Task::(resourceAcquired) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2022::DEBUG::2012-08-30 13:09:12,900::task::978::TaskManager.Task::(_decref) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::ref 1 aborting False
>Thread-2022::INFO::2012-08-30 13:09:12,905::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2022::DEBUG::2012-08-30 13:09:12,905::task::1172::TaskManager.Task::(prepare) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2022::DEBUG::2012-08-30 13:09:12,905::task::588::TaskManager.Task::(_updateState) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::moving from state preparing -> state finished
>Thread-2022::DEBUG::2012-08-30 13:09:12,905::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2022::DEBUG::2012-08-30 13:09:12,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2022::DEBUG::2012-08-30 13:09:12,906::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2022::DEBUG::2012-08-30 13:09:12,906::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2022::DEBUG::2012-08-30 13:09:12,907::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2022::DEBUG::2012-08-30 13:09:12,907::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2022::DEBUG::2012-08-30 13:09:12,907::task::978::TaskManager.Task::(_decref) Task=`6ca84866-fd26-4ee1-a99e-360914aee68d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:09:13,532::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:15,662::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:15,662::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2024::DEBUG::2012-08-30 13:09:16,075::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2024::DEBUG::2012-08-30 13:09:16,075::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:09:18,533::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:20,663::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:23,534::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:23,535::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:25,665::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:09:25,993::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2026::DEBUG::2012-08-30 13:09:26,178::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2026::DEBUG::2012-08-30 13:09:26,178::task::588::TaskManager.Task::(_updateState) Task=`1d31158c-78f0-4688-9576-0e703051b93b`::moving from state init -> state preparing
>Thread-2026::INFO::2012-08-30 13:09:26,178::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2026::INFO::2012-08-30 13:09:26,179::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2026::DEBUG::2012-08-30 13:09:26,179::task::1172::TaskManager.Task::(prepare) Task=`1d31158c-78f0-4688-9576-0e703051b93b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2026::DEBUG::2012-08-30 13:09:26,179::task::588::TaskManager.Task::(_updateState) Task=`1d31158c-78f0-4688-9576-0e703051b93b`::moving from state preparing -> state finished
>Thread-2026::DEBUG::2012-08-30 13:09:26,180::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2026::DEBUG::2012-08-30 13:09:26,180::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2026::DEBUG::2012-08-30 13:09:26,180::task::978::TaskManager.Task::(_decref) Task=`1d31158c-78f0-4688-9576-0e703051b93b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:09:28,536::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2028::DEBUG::2012-08-30 13:09:29,413::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2028::DEBUG::2012-08-30 13:09:29,414::task::588::TaskManager.Task::(_updateState) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::moving from state init -> state preparing
>Thread-2028::INFO::2012-08-30 13:09:29,414::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2028::DEBUG::2012-08-30 13:09:29,414::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`78309f2c-f36e-4514-8735-d03ebec19a64`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2028::DEBUG::2012-08-30 13:09:29,414::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2028::DEBUG::2012-08-30 13:09:29,415::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2028::DEBUG::2012-08-30 13:09:29,415::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`78309f2c-f36e-4514-8735-d03ebec19a64`::Granted request
>Thread-2028::DEBUG::2012-08-30 13:09:29,415::task::817::TaskManager.Task::(resourceAcquired) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2028::DEBUG::2012-08-30 13:09:29,416::task::978::TaskManager.Task::(_decref) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::ref 1 aborting False
>Thread-2028::INFO::2012-08-30 13:09:29,418::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2028::DEBUG::2012-08-30 13:09:29,419::task::1172::TaskManager.Task::(prepare) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2028::DEBUG::2012-08-30 13:09:29,419::task::588::TaskManager.Task::(_updateState) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::moving from state preparing -> state finished
>Thread-2028::DEBUG::2012-08-30 13:09:29,419::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2028::DEBUG::2012-08-30 13:09:29,419::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2028::DEBUG::2012-08-30 13:09:29,420::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2028::DEBUG::2012-08-30 13:09:29,420::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2028::DEBUG::2012-08-30 13:09:29,420::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2028::DEBUG::2012-08-30 13:09:29,420::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2028::DEBUG::2012-08-30 13:09:29,421::task::978::TaskManager.Task::(_decref) Task=`9cb1f178-23a7-4089-a5bd-de972b8c990b`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 13:09:30,666::task::588::TaskManager.Task::(_updateState) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:09:30,666::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:09:30,666::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`3f8eb9ad-1d2a-46ea-aeb6-fbdace8cc8f1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:09:30,667::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:09:30,667::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:09:30,667::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`3f8eb9ad-1d2a-46ea-aeb6-fbdace8cc8f1`::Granted request
>Thread-104::DEBUG::2012-08-30 13:09:30,667::task::817::TaskManager.Task::(resourceAcquired) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:09:30,668::task::978::TaskManager.Task::(_decref) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:09:30,669::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:09:30,671::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:09:30,672::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:09:30,673::task::1172::TaskManager.Task::(prepare) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:09:30,673::task::588::TaskManager.Task::(_updateState) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:09:30,673::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:09:30,673::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:09:30,674::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:09:30,674::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:09:30,674::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:09:30,674::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:09:30,675::task::978::TaskManager.Task::(_decref) Task=`a06141c8-df41-4f6a-8e66-3c44a2767328`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:09:30,675::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:30,675::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:30,676::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:30,676::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:09:33,000::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:09:33,537::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:35,677::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:38,537::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:38,538::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:40,678::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2031::DEBUG::2012-08-30 13:09:41,596::task::588::TaskManager.Task::(_updateState) Task=`6ccf8132-21e3-4298-ad0a-9b9ec81e2654`::moving from state init -> state preparing
>Thread-2031::INFO::2012-08-30 13:09:41,597::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2031::INFO::2012-08-30 13:09:41,597::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161385536194', 'lastCheck': 1346303380.7674611, 'code': 0, 'valid': True}}
>Thread-2031::DEBUG::2012-08-30 13:09:41,597::task::1172::TaskManager.Task::(prepare) Task=`6ccf8132-21e3-4298-ad0a-9b9ec81e2654`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00161385536194', 'lastCheck': 1346303380.7674611, 'code': 0, 'valid': True}}
>Thread-2031::DEBUG::2012-08-30 13:09:41,597::task::588::TaskManager.Task::(_updateState) Task=`6ccf8132-21e3-4298-ad0a-9b9ec81e2654`::moving from state preparing -> state finished
>Thread-2031::DEBUG::2012-08-30 13:09:41,597::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2031::DEBUG::2012-08-30 13:09:41,598::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2031::DEBUG::2012-08-30 13:09:41,598::task::978::TaskManager.Task::(_decref) Task=`6ccf8132-21e3-4298-ad0a-9b9ec81e2654`::ref 0 aborting False
>Thread-2032::DEBUG::2012-08-30 13:09:43,163::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2032::DEBUG::2012-08-30 13:09:43,164::task::588::TaskManager.Task::(_updateState) Task=`4d377e3d-3d81-4597-93ac-41ee1bd43042`::moving from state init -> state preparing
>Thread-2032::INFO::2012-08-30 13:09:43,164::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2032::INFO::2012-08-30 13:09:43,164::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2032::DEBUG::2012-08-30 13:09:43,164::task::1172::TaskManager.Task::(prepare) Task=`4d377e3d-3d81-4597-93ac-41ee1bd43042`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2032::DEBUG::2012-08-30 13:09:43,165::task::588::TaskManager.Task::(_updateState) Task=`4d377e3d-3d81-4597-93ac-41ee1bd43042`::moving from state preparing -> state finished
>Thread-2032::DEBUG::2012-08-30 13:09:43,165::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2032::DEBUG::2012-08-30 13:09:43,165::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2032::DEBUG::2012-08-30 13:09:43,165::task::978::TaskManager.Task::(_decref) Task=`4d377e3d-3d81-4597-93ac-41ee1bd43042`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:09:43,539::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2033::DEBUG::2012-08-30 13:09:44,723::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2033::DEBUG::2012-08-30 13:09:44,723::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:09:45,680::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:45,680::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2034::DEBUG::2012-08-30 13:09:46,289::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2034::DEBUG::2012-08-30 13:09:46,289::task::588::TaskManager.Task::(_updateState) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::moving from state init -> state preparing
>Thread-2034::INFO::2012-08-30 13:09:46,289::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2034::DEBUG::2012-08-30 13:09:46,290::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`173bf5dd-0a3a-4e4c-8d6d-d464e496c174`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2034::DEBUG::2012-08-30 13:09:46,290::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2034::DEBUG::2012-08-30 13:09:46,290::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2034::DEBUG::2012-08-30 13:09:46,290::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`173bf5dd-0a3a-4e4c-8d6d-d464e496c174`::Granted request
>Thread-2034::DEBUG::2012-08-30 13:09:46,291::task::817::TaskManager.Task::(resourceAcquired) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2034::DEBUG::2012-08-30 13:09:46,291::task::978::TaskManager.Task::(_decref) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::ref 1 aborting False
>Thread-2034::INFO::2012-08-30 13:09:46,294::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2034::DEBUG::2012-08-30 13:09:46,294::task::1172::TaskManager.Task::(prepare) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2034::DEBUG::2012-08-30 13:09:46,294::task::588::TaskManager.Task::(_updateState) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::moving from state preparing -> state finished
>Thread-2034::DEBUG::2012-08-30 13:09:46,295::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2034::DEBUG::2012-08-30 13:09:46,295::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2034::DEBUG::2012-08-30 13:09:46,295::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2034::DEBUG::2012-08-30 13:09:46,295::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2034::DEBUG::2012-08-30 13:09:46,296::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2034::DEBUG::2012-08-30 13:09:46,296::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2034::DEBUG::2012-08-30 13:09:46,296::task::978::TaskManager.Task::(_decref) Task=`b7dd902d-f662-4a9c-a696-ef27c321d641`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:09:48,540::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:50,682::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:09:53,541::task::588::TaskManager.Task::(_updateState) Task=`be20643a-e938-4378-982a-0fe69b075d56`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:09:53,541::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:09:53,542::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`333a414e-7712-4e24-a688-88147fe52e1d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:09:53,542::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:09:53,542::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:09:53,542::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`333a414e-7712-4e24-a688-88147fe52e1d`::Granted request
>Thread-40::DEBUG::2012-08-30 13:09:53,543::task::817::TaskManager.Task::(resourceAcquired) Task=`be20643a-e938-4378-982a-0fe69b075d56`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:09:53,543::task::978::TaskManager.Task::(_decref) Task=`be20643a-e938-4378-982a-0fe69b075d56`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:09:53,545::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:09:53,547::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:09:53,548::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:09:53,548::task::1172::TaskManager.Task::(prepare) Task=`be20643a-e938-4378-982a-0fe69b075d56`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:09:53,548::task::588::TaskManager.Task::(_updateState) Task=`be20643a-e938-4378-982a-0fe69b075d56`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:09:53,549::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:09:53,549::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:09:53,549::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:09:53,549::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:09:53,550::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:09:53,550::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:09:53,550::task::978::TaskManager.Task::(_decref) Task=`be20643a-e938-4378-982a-0fe69b075d56`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:09:53,550::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:53,551::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:53,551::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:09:53,552::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:09:55,683::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:09:56,023::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:09:58,553::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2037::DEBUG::2012-08-30 13:09:59,509::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2037::DEBUG::2012-08-30 13:09:59,509::task::588::TaskManager.Task::(_updateState) Task=`0567c659-5b94-412c-8282-69f9d03c88b3`::moving from state init -> state preparing
>Thread-2037::INFO::2012-08-30 13:09:59,509::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2037::INFO::2012-08-30 13:09:59,510::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2037::DEBUG::2012-08-30 13:09:59,510::task::1172::TaskManager.Task::(prepare) Task=`0567c659-5b94-412c-8282-69f9d03c88b3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2037::DEBUG::2012-08-30 13:09:59,510::task::588::TaskManager.Task::(_updateState) Task=`0567c659-5b94-412c-8282-69f9d03c88b3`::moving from state preparing -> state finished
>Thread-2037::DEBUG::2012-08-30 13:09:59,510::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2037::DEBUG::2012-08-30 13:09:59,511::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2037::DEBUG::2012-08-30 13:09:59,511::task::978::TaskManager.Task::(_decref) Task=`0567c659-5b94-412c-8282-69f9d03c88b3`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:10:00,684::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:00,685::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2039::DEBUG::2012-08-30 13:10:02,652::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2039::DEBUG::2012-08-30 13:10:02,652::task::588::TaskManager.Task::(_updateState) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::moving from state init -> state preparing
>Thread-2039::INFO::2012-08-30 13:10:02,653::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2039::DEBUG::2012-08-30 13:10:02,653::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`51048185-3e73-475f-a68f-c733d8b2086e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2039::DEBUG::2012-08-30 13:10:02,653::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2039::DEBUG::2012-08-30 13:10:02,654::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2039::DEBUG::2012-08-30 13:10:02,654::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`51048185-3e73-475f-a68f-c733d8b2086e`::Granted request
>Thread-2039::DEBUG::2012-08-30 13:10:02,654::task::817::TaskManager.Task::(resourceAcquired) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2039::DEBUG::2012-08-30 13:10:02,654::task::978::TaskManager.Task::(_decref) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::ref 1 aborting False
>Thread-2039::INFO::2012-08-30 13:10:02,658::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2039::DEBUG::2012-08-30 13:10:02,658::task::1172::TaskManager.Task::(prepare) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2039::DEBUG::2012-08-30 13:10:02,658::task::588::TaskManager.Task::(_updateState) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::moving from state preparing -> state finished
>Thread-2039::DEBUG::2012-08-30 13:10:02,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2039::DEBUG::2012-08-30 13:10:02,659::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2039::DEBUG::2012-08-30 13:10:02,659::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2039::DEBUG::2012-08-30 13:10:02,659::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2039::DEBUG::2012-08-30 13:10:02,659::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2039::DEBUG::2012-08-30 13:10:02,660::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2039::DEBUG::2012-08-30 13:10:02,660::task::978::TaskManager.Task::(_decref) Task=`c501409f-0c0c-43da-b6c0-0ec1475c8c8f`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:10:03,030::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:10:03,554::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:05,686::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:08,555::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:08,555::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2041::DEBUG::2012-08-30 13:10:10,615::task::588::TaskManager.Task::(_updateState) Task=`7c96cc23-a5c0-4e7b-b1eb-a6206a156b6c`::moving from state init -> state preparing
>Thread-2041::INFO::2012-08-30 13:10:10,615::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2041::INFO::2012-08-30 13:10:10,616::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177717208862', 'lastCheck': 1346303400.7816091, 'code': 0, 'valid': True}}
>Thread-2041::DEBUG::2012-08-30 13:10:10,616::task::1172::TaskManager.Task::(prepare) Task=`7c96cc23-a5c0-4e7b-b1eb-a6206a156b6c`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177717208862', 'lastCheck': 1346303400.7816091, 'code': 0, 'valid': True}}
>Thread-2041::DEBUG::2012-08-30 13:10:10,616::task::588::TaskManager.Task::(_updateState) Task=`7c96cc23-a5c0-4e7b-b1eb-a6206a156b6c`::moving from state preparing -> state finished
>Thread-2041::DEBUG::2012-08-30 13:10:10,616::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2041::DEBUG::2012-08-30 13:10:10,617::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2041::DEBUG::2012-08-30 13:10:10,617::task::978::TaskManager.Task::(_decref) Task=`7c96cc23-a5c0-4e7b-b1eb-a6206a156b6c`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:10:10,686::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:13,556::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2042::DEBUG::2012-08-30 13:10:13,782::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2042::DEBUG::2012-08-30 13:10:13,782::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-2043::DEBUG::2012-08-30 13:10:15,348::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2043::DEBUG::2012-08-30 13:10:15,349::task::588::TaskManager.Task::(_updateState) Task=`b2cf501b-0753-44bc-9ce1-c8518da0a80b`::moving from state init -> state preparing
>Thread-2043::INFO::2012-08-30 13:10:15,349::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2043::INFO::2012-08-30 13:10:15,349::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2043::DEBUG::2012-08-30 13:10:15,350::task::1172::TaskManager.Task::(prepare) Task=`b2cf501b-0753-44bc-9ce1-c8518da0a80b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2043::DEBUG::2012-08-30 13:10:15,350::task::588::TaskManager.Task::(_updateState) Task=`b2cf501b-0753-44bc-9ce1-c8518da0a80b`::moving from state preparing -> state finished
>Thread-2043::DEBUG::2012-08-30 13:10:15,350::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2043::DEBUG::2012-08-30 13:10:15,350::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2043::DEBUG::2012-08-30 13:10:15,351::task::978::TaskManager.Task::(_decref) Task=`b2cf501b-0753-44bc-9ce1-c8518da0a80b`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:10:15,687::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:15,688::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2044::DEBUG::2012-08-30 13:10:18,512::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2044::DEBUG::2012-08-30 13:10:18,512::task::588::TaskManager.Task::(_updateState) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::moving from state init -> state preparing
>Thread-2044::INFO::2012-08-30 13:10:18,512::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2044::DEBUG::2012-08-30 13:10:18,513::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`888e3066-7967-48e0-8819-71dc23bb8e4d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2044::DEBUG::2012-08-30 13:10:18,513::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2044::DEBUG::2012-08-30 13:10:18,513::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2044::DEBUG::2012-08-30 13:10:18,514::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`888e3066-7967-48e0-8819-71dc23bb8e4d`::Granted request
>Thread-2044::DEBUG::2012-08-30 13:10:18,514::task::817::TaskManager.Task::(resourceAcquired) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2044::DEBUG::2012-08-30 13:10:18,514::task::978::TaskManager.Task::(_decref) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::ref 1 aborting False
>Thread-2044::INFO::2012-08-30 13:10:18,518::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2044::DEBUG::2012-08-30 13:10:18,518::task::1172::TaskManager.Task::(prepare) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2044::DEBUG::2012-08-30 13:10:18,518::task::588::TaskManager.Task::(_updateState) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::moving from state preparing -> state finished
>Thread-2044::DEBUG::2012-08-30 13:10:18,519::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2044::DEBUG::2012-08-30 13:10:18,519::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2044::DEBUG::2012-08-30 13:10:18,519::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2044::DEBUG::2012-08-30 13:10:18,519::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2044::DEBUG::2012-08-30 13:10:18,520::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2044::DEBUG::2012-08-30 13:10:18,520::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2044::DEBUG::2012-08-30 13:10:18,520::task::978::TaskManager.Task::(_decref) Task=`79fd9f79-5eb5-416c-b5a8-beb13a9655cc`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:10:18,558::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:20,689::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:23,559::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:23,560::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:25,690::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:10:26,051::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:10:28,561::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:10:30,691::task::588::TaskManager.Task::(_updateState) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:10:30,691::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:10:30,691::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`33ba124f-5630-4922-87b3-90bf952b0e34`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:10:30,692::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:10:30,692::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:10:30,692::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`33ba124f-5630-4922-87b3-90bf952b0e34`::Granted request
>Thread-104::DEBUG::2012-08-30 13:10:30,692::task::817::TaskManager.Task::(resourceAcquired) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:10:30,693::task::978::TaskManager.Task::(_decref) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:10:30,694::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:10:30,696::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:10:30,697::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:10:30,698::task::1172::TaskManager.Task::(prepare) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:10:30,698::task::588::TaskManager.Task::(_updateState) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:10:30,698::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:10:30,698::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:10:30,699::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:10:30,699::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:10:30,699::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:10:30,699::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:10:30,700::task::978::TaskManager.Task::(_decref) Task=`e5c9e2a2-998b-4195-a662-cb22171f0757`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:10:30,700::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:30,700::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:30,701::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:30,701::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-26::DEBUG::2012-08-30 13:10:30,794::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:10:30,794::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-26::DEBUG::2012-08-30 13:10:30,795::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None)
>Thread-26::DEBUG::2012-08-30 13:10:30,905::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = '  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5
>Thread-26::WARNING::2012-08-30 13:10:30,906::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found']
>Thread-26::DEBUG::2012-08-30 13:10:30,906::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-26::DEBUG::2012-08-30 13:10:30,912::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:10:30,913::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend
>Thread-26::DEBUG::2012-08-30 13:10:30,915::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d']
>Thread-26::DEBUG::2012-08-30 13:10:30,916::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-26::WARNING::2012-08-30 13:10:30,916::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered
>Thread-26::WARNING::2012-08-30 13:10:30,917::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered
>Thread-2048::DEBUG::2012-08-30 13:10:32,155::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2048::DEBUG::2012-08-30 13:10:32,155::task::588::TaskManager.Task::(_updateState) Task=`1e524eb9-c3a1-4a7b-b1d3-d7719800d9cf`::moving from state init -> state preparing
>Thread-2048::INFO::2012-08-30 13:10:32,155::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2048::INFO::2012-08-30 13:10:32,156::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2048::DEBUG::2012-08-30 13:10:32,156::task::1172::TaskManager.Task::(prepare) Task=`1e524eb9-c3a1-4a7b-b1d3-d7719800d9cf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2048::DEBUG::2012-08-30 13:10:32,156::task::588::TaskManager.Task::(_updateState) Task=`1e524eb9-c3a1-4a7b-b1d3-d7719800d9cf`::moving from state preparing -> state finished
>Thread-2048::DEBUG::2012-08-30 13:10:32,156::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2048::DEBUG::2012-08-30 13:10:32,157::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2048::DEBUG::2012-08-30 13:10:32,157::task::978::TaskManager.Task::(_decref) Task=`1e524eb9-c3a1-4a7b-b1d3-d7719800d9cf`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:10:33,058::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:10:33,563::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:35,702::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2055::DEBUG::2012-08-30 13:10:36,274::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2055::DEBUG::2012-08-30 13:10:36,275::task::588::TaskManager.Task::(_updateState) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::moving from state init -> state preparing
>Thread-2055::INFO::2012-08-30 13:10:36,275::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2055::DEBUG::2012-08-30 13:10:36,275::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`37a0c218-4f6e-49f2-92e0-42bcd3b5fb4a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2055::DEBUG::2012-08-30 13:10:36,276::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2055::DEBUG::2012-08-30 13:10:36,276::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2055::DEBUG::2012-08-30 13:10:36,276::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`37a0c218-4f6e-49f2-92e0-42bcd3b5fb4a`::Granted request
>Thread-2055::DEBUG::2012-08-30 13:10:36,276::task::817::TaskManager.Task::(resourceAcquired) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2055::DEBUG::2012-08-30 13:10:36,277::task::978::TaskManager.Task::(_decref) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::ref 1 aborting False
>Thread-2055::INFO::2012-08-30 13:10:36,280::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2055::DEBUG::2012-08-30 13:10:36,280::task::1172::TaskManager.Task::(prepare) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2055::DEBUG::2012-08-30 13:10:36,280::task::588::TaskManager.Task::(_updateState) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::moving from state preparing -> state finished
>Thread-2055::DEBUG::2012-08-30 13:10:36,280::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2055::DEBUG::2012-08-30 13:10:36,281::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2055::DEBUG::2012-08-30 13:10:36,281::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2055::DEBUG::2012-08-30 13:10:36,281::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2055::DEBUG::2012-08-30 13:10:36,281::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2055::DEBUG::2012-08-30 13:10:36,282::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2055::DEBUG::2012-08-30 13:10:36,282::task::978::TaskManager.Task::(_decref) Task=`e9d4b94e-392b-425d-95f1-4118dfa444b1`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:10:38,564::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:38,565::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2056::DEBUG::2012-08-30 13:10:39,758::task::588::TaskManager.Task::(_updateState) Task=`5ee3b4e7-8e74-418c-9ad5-e3fd332418b1`::moving from state init -> state preparing
>Thread-2056::INFO::2012-08-30 13:10:39,759::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2056::INFO::2012-08-30 13:10:39,759::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00157690048218', 'lastCheck': 1346303430.9232769, 'code': 0, 'valid': True}}
>Thread-2056::DEBUG::2012-08-30 13:10:39,759::task::1172::TaskManager.Task::(prepare) Task=`5ee3b4e7-8e74-418c-9ad5-e3fd332418b1`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00157690048218', 'lastCheck': 1346303430.9232769, 'code': 0, 'valid': True}}
>Thread-2056::DEBUG::2012-08-30 13:10:39,759::task::588::TaskManager.Task::(_updateState) Task=`5ee3b4e7-8e74-418c-9ad5-e3fd332418b1`::moving from state preparing -> state finished
>Thread-2056::DEBUG::2012-08-30 13:10:39,760::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2056::DEBUG::2012-08-30 13:10:39,760::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2056::DEBUG::2012-08-30 13:10:39,760::task::978::TaskManager.Task::(_decref) Task=`5ee3b4e7-8e74-418c-9ad5-e3fd332418b1`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:10:40,703::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2057::DEBUG::2012-08-30 13:10:43,403::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2057::DEBUG::2012-08-30 13:10:43,403::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:10:43,566::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:45,704::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:45,704::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:48,567::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2059::DEBUG::2012-08-30 13:10:49,508::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2059::DEBUG::2012-08-30 13:10:49,508::task::588::TaskManager.Task::(_updateState) Task=`fcefc378-6b99-4038-845d-eb025e11adfa`::moving from state init -> state preparing
>Thread-2059::INFO::2012-08-30 13:10:49,509::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2059::INFO::2012-08-30 13:10:49,509::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2059::DEBUG::2012-08-30 13:10:49,509::task::1172::TaskManager.Task::(prepare) Task=`fcefc378-6b99-4038-845d-eb025e11adfa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2059::DEBUG::2012-08-30 13:10:49,509::task::588::TaskManager.Task::(_updateState) Task=`fcefc378-6b99-4038-845d-eb025e11adfa`::moving from state preparing -> state finished
>Thread-2059::DEBUG::2012-08-30 13:10:49,510::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2059::DEBUG::2012-08-30 13:10:49,510::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2059::DEBUG::2012-08-30 13:10:49,510::task::978::TaskManager.Task::(_decref) Task=`fcefc378-6b99-4038-845d-eb025e11adfa`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:10:50,705::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2060::DEBUG::2012-08-30 13:10:53,512::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2060::DEBUG::2012-08-30 13:10:53,513::task::588::TaskManager.Task::(_updateState) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::moving from state init -> state preparing
>Thread-2060::INFO::2012-08-30 13:10:53,513::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2060::DEBUG::2012-08-30 13:10:53,513::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0ee072a9-3928-48be-9809-1d1774f8c7c7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2060::DEBUG::2012-08-30 13:10:53,513::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2060::DEBUG::2012-08-30 13:10:53,514::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2060::DEBUG::2012-08-30 13:10:53,514::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0ee072a9-3928-48be-9809-1d1774f8c7c7`::Granted request
>Thread-2060::DEBUG::2012-08-30 13:10:53,515::task::817::TaskManager.Task::(resourceAcquired) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2060::DEBUG::2012-08-30 13:10:53,515::task::978::TaskManager.Task::(_decref) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::ref 1 aborting False
>Thread-2060::INFO::2012-08-30 13:10:53,518::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2060::DEBUG::2012-08-30 13:10:53,518::task::1172::TaskManager.Task::(prepare) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2060::DEBUG::2012-08-30 13:10:53,519::task::588::TaskManager.Task::(_updateState) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::moving from state preparing -> state finished
>Thread-2060::DEBUG::2012-08-30 13:10:53,519::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2060::DEBUG::2012-08-30 13:10:53,519::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2060::DEBUG::2012-08-30 13:10:53,519::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2060::DEBUG::2012-08-30 13:10:53,520::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2060::DEBUG::2012-08-30 13:10:53,520::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2060::DEBUG::2012-08-30 13:10:53,520::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2060::DEBUG::2012-08-30 13:10:53,520::task::978::TaskManager.Task::(_decref) Task=`33a777be-895c-4ab4-bc7a-e7bd2eea4181`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 13:10:53,568::task::588::TaskManager.Task::(_updateState) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:10:53,568::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:10:53,569::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`06cfd44a-2137-457b-9a60-75b6936bdf83`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:10:53,569::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:10:53,569::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:10:53,569::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`06cfd44a-2137-457b-9a60-75b6936bdf83`::Granted request
>Thread-40::DEBUG::2012-08-30 13:10:53,570::task::817::TaskManager.Task::(resourceAcquired) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:10:53,570::task::978::TaskManager.Task::(_decref) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:10:53,571::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:10:53,574::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:10:53,575::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:10:53,575::task::1172::TaskManager.Task::(prepare) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:10:53,575::task::588::TaskManager.Task::(_updateState) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:10:53,575::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:10:53,576::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:10:53,576::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:10:53,576::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:10:53,576::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:10:53,577::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:10:53,577::task::978::TaskManager.Task::(_decref) Task=`c68040a2-4d0d-4d4d-9980-3ef99f2c9c48`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:10:53,577::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:53,578::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:53,578::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:10:53,578::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:10:55,706::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:10:56,080::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:10:58,580::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:00,707::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:00,708::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:11:03,087::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:11:03,581::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:05,709::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2064::DEBUG::2012-08-30 13:11:06,694::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2064::DEBUG::2012-08-30 13:11:06,694::task::588::TaskManager.Task::(_updateState) Task=`67ce3f01-e38a-4df2-adf9-fcc740ee133e`::moving from state init -> state preparing
>Thread-2064::INFO::2012-08-30 13:11:06,694::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2064::INFO::2012-08-30 13:11:06,695::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2064::DEBUG::2012-08-30 13:11:06,695::task::1172::TaskManager.Task::(prepare) Task=`67ce3f01-e38a-4df2-adf9-fcc740ee133e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2064::DEBUG::2012-08-30 13:11:06,695::task::588::TaskManager.Task::(_updateState) Task=`67ce3f01-e38a-4df2-adf9-fcc740ee133e`::moving from state preparing -> state finished
>Thread-2064::DEBUG::2012-08-30 13:11:06,695::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2064::DEBUG::2012-08-30 13:11:06,696::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2064::DEBUG::2012-08-30 13:11:06,696::task::978::TaskManager.Task::(_decref) Task=`67ce3f01-e38a-4df2-adf9-fcc740ee133e`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:11:08,582::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:08,583::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2065::DEBUG::2012-08-30 13:11:09,087::task::588::TaskManager.Task::(_updateState) Task=`becf2e26-3bee-4db5-854b-836e5169c3ec`::moving from state init -> state preparing
>Thread-2065::INFO::2012-08-30 13:11:09,088::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2065::INFO::2012-08-30 13:11:09,088::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177311897278', 'lastCheck': 1346303460.9437189, 'code': 0, 'valid': True}}
>Thread-2065::DEBUG::2012-08-30 13:11:09,088::task::1172::TaskManager.Task::(prepare) Task=`becf2e26-3bee-4db5-854b-836e5169c3ec`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177311897278', 'lastCheck': 1346303460.9437189, 'code': 0, 'valid': True}}
>Thread-2065::DEBUG::2012-08-30 13:11:09,088::task::588::TaskManager.Task::(_updateState) Task=`becf2e26-3bee-4db5-854b-836e5169c3ec`::moving from state preparing -> state finished
>Thread-2065::DEBUG::2012-08-30 13:11:09,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2065::DEBUG::2012-08-30 13:11:09,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2065::DEBUG::2012-08-30 13:11:09,089::task::978::TaskManager.Task::(_decref) Task=`becf2e26-3bee-4db5-854b-836e5169c3ec`::ref 0 aborting False
>Thread-2066::DEBUG::2012-08-30 13:11:09,845::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2066::DEBUG::2012-08-30 13:11:09,846::task::588::TaskManager.Task::(_updateState) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::moving from state init -> state preparing
>Thread-2066::INFO::2012-08-30 13:11:09,846::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2066::DEBUG::2012-08-30 13:11:09,846::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3d56dc96-20bd-4a42-8802-3743dbb3411a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2066::DEBUG::2012-08-30 13:11:09,846::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2066::DEBUG::2012-08-30 13:11:09,847::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2066::DEBUG::2012-08-30 13:11:09,847::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`3d56dc96-20bd-4a42-8802-3743dbb3411a`::Granted request
>Thread-2066::DEBUG::2012-08-30 13:11:09,847::task::817::TaskManager.Task::(resourceAcquired) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2066::DEBUG::2012-08-30 13:11:09,847::task::978::TaskManager.Task::(_decref) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::ref 1 aborting False
>Thread-2066::INFO::2012-08-30 13:11:09,850::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2066::DEBUG::2012-08-30 13:11:09,851::task::1172::TaskManager.Task::(prepare) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2066::DEBUG::2012-08-30 13:11:09,851::task::588::TaskManager.Task::(_updateState) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::moving from state preparing -> state finished
>Thread-2066::DEBUG::2012-08-30 13:11:09,851::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2066::DEBUG::2012-08-30 13:11:09,851::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2066::DEBUG::2012-08-30 13:11:09,852::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2066::DEBUG::2012-08-30 13:11:09,852::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2066::DEBUG::2012-08-30 13:11:09,852::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2066::DEBUG::2012-08-30 13:11:09,852::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2066::DEBUG::2012-08-30 13:11:09,853::task::978::TaskManager.Task::(_decref) Task=`3ac2e8e1-e29f-4d49-93b7-8d7a8f38254e`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:11:10,710::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2067::DEBUG::2012-08-30 13:11:12,245::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2067::DEBUG::2012-08-30 13:11:12,245::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:11:13,584::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:15,711::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:15,711::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:18,585::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:20,712::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:23,587::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:23,587::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2070::DEBUG::2012-08-30 13:11:23,854::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2070::DEBUG::2012-08-30 13:11:23,854::task::588::TaskManager.Task::(_updateState) Task=`bb714a8b-cb94-4f55-90d8-3ce27664fded`::moving from state init -> state preparing
>Thread-2070::INFO::2012-08-30 13:11:23,855::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2070::INFO::2012-08-30 13:11:23,855::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2070::DEBUG::2012-08-30 13:11:23,855::task::1172::TaskManager.Task::(prepare) Task=`bb714a8b-cb94-4f55-90d8-3ce27664fded`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2070::DEBUG::2012-08-30 13:11:23,856::task::588::TaskManager.Task::(_updateState) Task=`bb714a8b-cb94-4f55-90d8-3ce27664fded`::moving from state preparing -> state finished
>Thread-2070::DEBUG::2012-08-30 13:11:23,856::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2070::DEBUG::2012-08-30 13:11:23,856::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2070::DEBUG::2012-08-30 13:11:23,856::task::978::TaskManager.Task::(_decref) Task=`bb714a8b-cb94-4f55-90d8-3ce27664fded`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:11:25,713::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:11:26,109::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2071::DEBUG::2012-08-30 13:11:27,532::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2071::DEBUG::2012-08-30 13:11:27,533::task::588::TaskManager.Task::(_updateState) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::moving from state init -> state preparing
>Thread-2071::INFO::2012-08-30 13:11:27,533::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2071::DEBUG::2012-08-30 13:11:27,533::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`da83211b-bc78-4b25-a88d-4b60870bfcba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2071::DEBUG::2012-08-30 13:11:27,533::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2071::DEBUG::2012-08-30 13:11:27,534::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2071::DEBUG::2012-08-30 13:11:27,534::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`da83211b-bc78-4b25-a88d-4b60870bfcba`::Granted request
>Thread-2071::DEBUG::2012-08-30 13:11:27,534::task::817::TaskManager.Task::(resourceAcquired) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2071::DEBUG::2012-08-30 13:11:27,535::task::978::TaskManager.Task::(_decref) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::ref 1 aborting False
>Thread-2071::INFO::2012-08-30 13:11:27,537::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2071::DEBUG::2012-08-30 13:11:27,538::task::1172::TaskManager.Task::(prepare) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2071::DEBUG::2012-08-30 13:11:27,538::task::588::TaskManager.Task::(_updateState) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::moving from state preparing -> state finished
>Thread-2071::DEBUG::2012-08-30 13:11:27,538::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2071::DEBUG::2012-08-30 13:11:27,538::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2071::DEBUG::2012-08-30 13:11:27,539::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2071::DEBUG::2012-08-30 13:11:27,539::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2071::DEBUG::2012-08-30 13:11:27,539::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2071::DEBUG::2012-08-30 13:11:27,539::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2071::DEBUG::2012-08-30 13:11:27,540::task::978::TaskManager.Task::(_decref) Task=`2e6f32e4-3b9d-4954-a1ee-5033fcfb56f4`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:11:28,588::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:11:30,714::task::588::TaskManager.Task::(_updateState) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:11:30,715::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:11:30,715::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d6646948-9c83-4d51-9d95-5c7c30d55581`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:11:30,715::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:11:30,715::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:11:30,716::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`d6646948-9c83-4d51-9d95-5c7c30d55581`::Granted request
>Thread-104::DEBUG::2012-08-30 13:11:30,716::task::817::TaskManager.Task::(resourceAcquired) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:11:30,716::task::978::TaskManager.Task::(_decref) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:11:30,718::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:11:30,720::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:11:30,721::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:11:30,721::task::1172::TaskManager.Task::(prepare) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:11:30,721::task::588::TaskManager.Task::(_updateState) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:11:30,722::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:11:30,722::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:11:30,722::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:11:30,722::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:11:30,723::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:11:30,723::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:11:30,723::task::978::TaskManager.Task::(_decref) Task=`1af08e5b-43b4-48cf-b0a5-53e4a1c23495`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:11:30,723::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:30,724::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:30,724::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:30,725::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:11:33,116::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:11:33,590::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:35,726::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:38,591::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:38,591::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2074::DEBUG::2012-08-30 13:11:39,160::task::588::TaskManager.Task::(_updateState) Task=`869c3305-a46d-466f-8b67-e2a2811f6035`::moving from state init -> state preparing
>Thread-2074::INFO::2012-08-30 13:11:39,160::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2074::INFO::2012-08-30 13:11:39,161::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346303490.9625361, 'code': 0, 'valid': True}}
>Thread-2074::DEBUG::2012-08-30 13:11:39,161::task::1172::TaskManager.Task::(prepare) Task=`869c3305-a46d-466f-8b67-e2a2811f6035`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346303490.9625361, 'code': 0, 'valid': True}}
>Thread-2074::DEBUG::2012-08-30 13:11:39,161::task::588::TaskManager.Task::(_updateState) Task=`869c3305-a46d-466f-8b67-e2a2811f6035`::moving from state preparing -> state finished
>Thread-2074::DEBUG::2012-08-30 13:11:39,161::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2074::DEBUG::2012-08-30 13:11:39,162::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2074::DEBUG::2012-08-30 13:11:39,162::task::978::TaskManager.Task::(_decref) Task=`869c3305-a46d-466f-8b67-e2a2811f6035`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:11:40,727::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2075::DEBUG::2012-08-30 13:11:40,754::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2075::DEBUG::2012-08-30 13:11:40,755::task::588::TaskManager.Task::(_updateState) Task=`a73088d1-2285-4be3-b578-909e85ba9749`::moving from state init -> state preparing
>Thread-2075::INFO::2012-08-30 13:11:40,755::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2075::INFO::2012-08-30 13:11:40,755::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2075::DEBUG::2012-08-30 13:11:40,756::task::1172::TaskManager.Task::(prepare) Task=`a73088d1-2285-4be3-b578-909e85ba9749`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2075::DEBUG::2012-08-30 13:11:40,756::task::588::TaskManager.Task::(_updateState) Task=`a73088d1-2285-4be3-b578-909e85ba9749`::moving from state preparing -> state finished
>Thread-2075::DEBUG::2012-08-30 13:11:40,756::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2075::DEBUG::2012-08-30 13:11:40,756::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2075::DEBUG::2012-08-30 13:11:40,756::task::978::TaskManager.Task::(_decref) Task=`a73088d1-2285-4be3-b578-909e85ba9749`::ref 0 aborting False
>Thread-2076::DEBUG::2012-08-30 13:11:42,334::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2076::DEBUG::2012-08-30 13:11:42,334::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:11:43,592::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2077::DEBUG::2012-08-30 13:11:43,999::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2077::DEBUG::2012-08-30 13:11:44,000::task::588::TaskManager.Task::(_updateState) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::moving from state init -> state preparing
>Thread-2077::INFO::2012-08-30 13:11:44,000::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2077::DEBUG::2012-08-30 13:11:44,000::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f288e760-4acb-4d2c-984a-875252619b48`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2077::DEBUG::2012-08-30 13:11:44,001::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2077::DEBUG::2012-08-30 13:11:44,001::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2077::DEBUG::2012-08-30 13:11:44,001::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f288e760-4acb-4d2c-984a-875252619b48`::Granted request
>Thread-2077::DEBUG::2012-08-30 13:11:44,001::task::817::TaskManager.Task::(resourceAcquired) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2077::DEBUG::2012-08-30 13:11:44,002::task::978::TaskManager.Task::(_decref) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::ref 1 aborting False
>Thread-2077::INFO::2012-08-30 13:11:44,005::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2077::DEBUG::2012-08-30 13:11:44,005::task::1172::TaskManager.Task::(prepare) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2077::DEBUG::2012-08-30 13:11:44,005::task::588::TaskManager.Task::(_updateState) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::moving from state preparing -> state finished
>Thread-2077::DEBUG::2012-08-30 13:11:44,005::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2077::DEBUG::2012-08-30 13:11:44,006::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2077::DEBUG::2012-08-30 13:11:44,006::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2077::DEBUG::2012-08-30 13:11:44,006::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2077::DEBUG::2012-08-30 13:11:44,006::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2077::DEBUG::2012-08-30 13:11:44,007::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2077::DEBUG::2012-08-30 13:11:44,007::task::978::TaskManager.Task::(_decref) Task=`9a087e2f-2941-46bb-906c-b3d6bce2d066`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:11:45,728::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:45,728::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:48,594::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:50,729::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:11:53,595::task::588::TaskManager.Task::(_updateState) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:11:53,595::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:11:53,596::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`10a218d6-7b5c-4415-94af-b87d9262e838`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:11:53,596::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:11:53,596::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:11:53,597::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`10a218d6-7b5c-4415-94af-b87d9262e838`::Granted request
>Thread-40::DEBUG::2012-08-30 13:11:53,597::task::817::TaskManager.Task::(resourceAcquired) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:11:53,597::task::978::TaskManager.Task::(_decref) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:11:53,599::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:11:53,601::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:11:53,602::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:11:53,602::task::1172::TaskManager.Task::(prepare) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:11:53,603::task::588::TaskManager.Task::(_updateState) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:11:53,603::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:11:53,603::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:11:53,603::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:11:53,604::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:11:53,604::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:11:53,604::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:11:53,604::task::978::TaskManager.Task::(_decref) Task=`53425c8e-3667-4037-b459-34f1ece1d4fa`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:11:53,605::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:53,605::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:53,606::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:11:53,606::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:11:55,730::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:11:56,134::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2080::DEBUG::2012-08-30 13:11:57,173::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2080::DEBUG::2012-08-30 13:11:57,173::task::588::TaskManager.Task::(_updateState) Task=`544ed687-8158-4eec-82c4-22ff4d454049`::moving from state init -> state preparing
>Thread-2080::INFO::2012-08-30 13:11:57,173::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2080::INFO::2012-08-30 13:11:57,174::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2080::DEBUG::2012-08-30 13:11:57,174::task::1172::TaskManager.Task::(prepare) Task=`544ed687-8158-4eec-82c4-22ff4d454049`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2080::DEBUG::2012-08-30 13:11:57,174::task::588::TaskManager.Task::(_updateState) Task=`544ed687-8158-4eec-82c4-22ff4d454049`::moving from state preparing -> state finished
>Thread-2080::DEBUG::2012-08-30 13:11:57,174::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2080::DEBUG::2012-08-30 13:11:57,175::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2080::DEBUG::2012-08-30 13:11:57,175::task::978::TaskManager.Task::(_decref) Task=`544ed687-8158-4eec-82c4-22ff4d454049`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:11:58,607::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2082::DEBUG::2012-08-30 13:12:00,335::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2082::DEBUG::2012-08-30 13:12:00,335::task::588::TaskManager.Task::(_updateState) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::moving from state init -> state preparing
>Thread-2082::INFO::2012-08-30 13:12:00,335::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2082::DEBUG::2012-08-30 13:12:00,336::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d8069c73-14d8-4421-9f23-a5b9944f0f32`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2082::DEBUG::2012-08-30 13:12:00,336::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2082::DEBUG::2012-08-30 13:12:00,336::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2082::DEBUG::2012-08-30 13:12:00,336::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d8069c73-14d8-4421-9f23-a5b9944f0f32`::Granted request
>Thread-2082::DEBUG::2012-08-30 13:12:00,337::task::817::TaskManager.Task::(resourceAcquired) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2082::DEBUG::2012-08-30 13:12:00,337::task::978::TaskManager.Task::(_decref) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::ref 1 aborting False
>Thread-2082::INFO::2012-08-30 13:12:00,340::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2082::DEBUG::2012-08-30 13:12:00,340::task::1172::TaskManager.Task::(prepare) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2082::DEBUG::2012-08-30 13:12:00,341::task::588::TaskManager.Task::(_updateState) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::moving from state preparing -> state finished
>Thread-2082::DEBUG::2012-08-30 13:12:00,341::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2082::DEBUG::2012-08-30 13:12:00,341::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2082::DEBUG::2012-08-30 13:12:00,341::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2082::DEBUG::2012-08-30 13:12:00,342::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2082::DEBUG::2012-08-30 13:12:00,342::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2082::DEBUG::2012-08-30 13:12:00,342::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2082::DEBUG::2012-08-30 13:12:00,342::task::978::TaskManager.Task::(_decref) Task=`095c9d67-8952-44f9-9844-e15de301ee62`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:12:00,731::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:00,732::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:12:03,140::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:12:03,608::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:05,732::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2084::DEBUG::2012-08-30 13:12:08,257::task::588::TaskManager.Task::(_updateState) Task=`661dff62-133b-4ab4-b21b-2a984aa974c8`::moving from state init -> state preparing
>Thread-2084::INFO::2012-08-30 13:12:08,257::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2084::INFO::2012-08-30 13:12:08,257::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177812576294', 'lastCheck': 1346303520.9828539, 'code': 0, 'valid': True}}
>Thread-2084::DEBUG::2012-08-30 13:12:08,258::task::1172::TaskManager.Task::(prepare) Task=`661dff62-133b-4ab4-b21b-2a984aa974c8`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177812576294', 'lastCheck': 1346303520.9828539, 'code': 0, 'valid': True}}
>Thread-2084::DEBUG::2012-08-30 13:12:08,258::task::588::TaskManager.Task::(_updateState) Task=`661dff62-133b-4ab4-b21b-2a984aa974c8`::moving from state preparing -> state finished
>Thread-2084::DEBUG::2012-08-30 13:12:08,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2084::DEBUG::2012-08-30 13:12:08,258::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2084::DEBUG::2012-08-30 13:12:08,259::task::978::TaskManager.Task::(_decref) Task=`661dff62-133b-4ab4-b21b-2a984aa974c8`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:12:08,609::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:08,610::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:10,733::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2085::DEBUG::2012-08-30 13:12:11,737::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2085::DEBUG::2012-08-30 13:12:11,738::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-2086::DEBUG::2012-08-30 13:12:13,307::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2086::DEBUG::2012-08-30 13:12:13,307::task::588::TaskManager.Task::(_updateState) Task=`9b2ede0b-1ea5-4bd8-ab14-4de7be082dc7`::moving from state init -> state preparing
>Thread-2086::INFO::2012-08-30 13:12:13,307::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2086::INFO::2012-08-30 13:12:13,308::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2086::DEBUG::2012-08-30 13:12:13,308::task::1172::TaskManager.Task::(prepare) Task=`9b2ede0b-1ea5-4bd8-ab14-4de7be082dc7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2086::DEBUG::2012-08-30 13:12:13,308::task::588::TaskManager.Task::(_updateState) Task=`9b2ede0b-1ea5-4bd8-ab14-4de7be082dc7`::moving from state preparing -> state finished
>Thread-2086::DEBUG::2012-08-30 13:12:13,308::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2086::DEBUG::2012-08-30 13:12:13,309::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2086::DEBUG::2012-08-30 13:12:13,309::task::978::TaskManager.Task::(_decref) Task=`9b2ede0b-1ea5-4bd8-ab14-4de7be082dc7`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:12:13,611::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:15,734::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:15,735::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2087::DEBUG::2012-08-30 13:12:16,457::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2087::DEBUG::2012-08-30 13:12:16,458::task::588::TaskManager.Task::(_updateState) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::moving from state init -> state preparing
>Thread-2087::INFO::2012-08-30 13:12:16,458::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2087::DEBUG::2012-08-30 13:12:16,458::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f7f6a7bf-3b8f-472a-b521-4424ae549db3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2087::DEBUG::2012-08-30 13:12:16,459::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2087::DEBUG::2012-08-30 13:12:16,459::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2087::DEBUG::2012-08-30 13:12:16,459::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f7f6a7bf-3b8f-472a-b521-4424ae549db3`::Granted request
>Thread-2087::DEBUG::2012-08-30 13:12:16,460::task::817::TaskManager.Task::(resourceAcquired) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2087::DEBUG::2012-08-30 13:12:16,460::task::978::TaskManager.Task::(_decref) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::ref 1 aborting False
>Thread-2087::INFO::2012-08-30 13:12:16,463::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2087::DEBUG::2012-08-30 13:12:16,463::task::1172::TaskManager.Task::(prepare) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2087::DEBUG::2012-08-30 13:12:16,463::task::588::TaskManager.Task::(_updateState) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::moving from state preparing -> state finished
>Thread-2087::DEBUG::2012-08-30 13:12:16,463::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2087::DEBUG::2012-08-30 13:12:16,464::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2087::DEBUG::2012-08-30 13:12:16,464::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2087::DEBUG::2012-08-30 13:12:16,464::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2087::DEBUG::2012-08-30 13:12:16,464::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2087::DEBUG::2012-08-30 13:12:16,465::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2087::DEBUG::2012-08-30 13:12:16,465::task::978::TaskManager.Task::(_decref) Task=`72cbfe39-eb75-46ac-a905-e03f195ce2c1`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:12:18,613::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:20,736::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:23,614::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:23,614::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:25,736::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:12:26,154::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:12:28,616::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2091::DEBUG::2012-08-30 13:12:29,638::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2091::DEBUG::2012-08-30 13:12:29,639::task::588::TaskManager.Task::(_updateState) Task=`5be5b960-b878-4c4d-82b5-47dbae1152de`::moving from state init -> state preparing
>Thread-2091::INFO::2012-08-30 13:12:29,639::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2091::INFO::2012-08-30 13:12:29,639::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2091::DEBUG::2012-08-30 13:12:29,639::task::1172::TaskManager.Task::(prepare) Task=`5be5b960-b878-4c4d-82b5-47dbae1152de`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2091::DEBUG::2012-08-30 13:12:29,640::task::588::TaskManager.Task::(_updateState) Task=`5be5b960-b878-4c4d-82b5-47dbae1152de`::moving from state preparing -> state finished
>Thread-2091::DEBUG::2012-08-30 13:12:29,640::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2091::DEBUG::2012-08-30 13:12:29,640::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2091::DEBUG::2012-08-30 13:12:29,640::task::978::TaskManager.Task::(_decref) Task=`5be5b960-b878-4c4d-82b5-47dbae1152de`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 13:12:30,737::task::588::TaskManager.Task::(_updateState) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:12:30,738::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:12:30,738::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bb2c0319-ef19-4827-a1ec-2ff661e12bc1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:12:30,738::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:12:30,739::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:12:30,739::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`bb2c0319-ef19-4827-a1ec-2ff661e12bc1`::Granted request
>Thread-104::DEBUG::2012-08-30 13:12:30,739::task::817::TaskManager.Task::(resourceAcquired) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:12:30,739::task::978::TaskManager.Task::(_decref) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:12:30,741::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:12:30,743::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:12:30,744::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:12:30,744::task::1172::TaskManager.Task::(prepare) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:12:30,745::task::588::TaskManager.Task::(_updateState) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:12:30,745::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:12:30,745::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:12:30,745::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:12:30,746::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:12:30,746::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:12:30,746::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:12:30,746::task::978::TaskManager.Task::(_decref) Task=`119d289e-3e19-4e7f-8ec0-17b53ada2595`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:12:30,747::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:30,747::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:30,748::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:30,748::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2093::DEBUG::2012-08-30 13:12:32,900::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2093::DEBUG::2012-08-30 13:12:32,900::task::588::TaskManager.Task::(_updateState) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::moving from state init -> state preparing
>Thread-2093::INFO::2012-08-30 13:12:32,901::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2093::DEBUG::2012-08-30 13:12:32,901::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c2d85981-8ba7-4315-8e1b-f70731fa336b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2093::DEBUG::2012-08-30 13:12:32,901::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2093::DEBUG::2012-08-30 13:12:32,901::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2093::DEBUG::2012-08-30 13:12:32,902::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c2d85981-8ba7-4315-8e1b-f70731fa336b`::Granted request
>Thread-2093::DEBUG::2012-08-30 13:12:32,902::task::817::TaskManager.Task::(resourceAcquired) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2093::DEBUG::2012-08-30 13:12:32,902::task::978::TaskManager.Task::(_decref) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::ref 1 aborting False
>Thread-2093::INFO::2012-08-30 13:12:32,905::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2093::DEBUG::2012-08-30 13:12:32,905::task::1172::TaskManager.Task::(prepare) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2093::DEBUG::2012-08-30 13:12:32,906::task::588::TaskManager.Task::(_updateState) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::moving from state preparing -> state finished
>Thread-2093::DEBUG::2012-08-30 13:12:32,906::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2093::DEBUG::2012-08-30 13:12:32,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2093::DEBUG::2012-08-30 13:12:32,906::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2093::DEBUG::2012-08-30 13:12:32,907::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2093::DEBUG::2012-08-30 13:12:32,907::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2093::DEBUG::2012-08-30 13:12:32,907::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2093::DEBUG::2012-08-30 13:12:32,907::task::978::TaskManager.Task::(_decref) Task=`5ad7e128-2712-4f75-8d3e-bb4f0c668bd1`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:12:33,160::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:12:33,617::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:35,749::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2094::DEBUG::2012-08-30 13:12:37,079::task::588::TaskManager.Task::(_updateState) Task=`87dfcd3a-1596-4ffb-9601-6cd89dab6a9c`::moving from state init -> state preparing
>Thread-2094::INFO::2012-08-30 13:12:37,079::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2094::INFO::2012-08-30 13:12:37,080::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162577629089', 'lastCheck': 1346303551.001581, 'code': 0, 'valid': True}}
>Thread-2094::DEBUG::2012-08-30 13:12:37,080::task::1172::TaskManager.Task::(prepare) Task=`87dfcd3a-1596-4ffb-9601-6cd89dab6a9c`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162577629089', 'lastCheck': 1346303551.001581, 'code': 0, 'valid': True}}
>Thread-2094::DEBUG::2012-08-30 13:12:37,080::task::588::TaskManager.Task::(_updateState) Task=`87dfcd3a-1596-4ffb-9601-6cd89dab6a9c`::moving from state preparing -> state finished
>Thread-2094::DEBUG::2012-08-30 13:12:37,080::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2094::DEBUG::2012-08-30 13:12:37,081::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2094::DEBUG::2012-08-30 13:12:37,081::task::978::TaskManager.Task::(_decref) Task=`87dfcd3a-1596-4ffb-9601-6cd89dab6a9c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:12:38,618::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:38,619::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2095::DEBUG::2012-08-30 13:12:40,231::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2095::DEBUG::2012-08-30 13:12:40,231::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:12:40,750::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:43,620::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2097::DEBUG::2012-08-30 13:12:45,462::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2097::DEBUG::2012-08-30 13:12:45,462::task::588::TaskManager.Task::(_updateState) Task=`58bb7e7d-14c2-46e6-bca3-f1c142aa1db1`::moving from state init -> state preparing
>Thread-2097::INFO::2012-08-30 13:12:45,462::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2097::INFO::2012-08-30 13:12:45,463::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2097::DEBUG::2012-08-30 13:12:45,463::task::1172::TaskManager.Task::(prepare) Task=`58bb7e7d-14c2-46e6-bca3-f1c142aa1db1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2097::DEBUG::2012-08-30 13:12:45,463::task::588::TaskManager.Task::(_updateState) Task=`58bb7e7d-14c2-46e6-bca3-f1c142aa1db1`::moving from state preparing -> state finished
>Thread-2097::DEBUG::2012-08-30 13:12:45,463::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2097::DEBUG::2012-08-30 13:12:45,464::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2097::DEBUG::2012-08-30 13:12:45,464::task::978::TaskManager.Task::(_decref) Task=`58bb7e7d-14c2-46e6-bca3-f1c142aa1db1`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:12:45,751::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:45,751::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:48,621::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2098::DEBUG::2012-08-30 13:12:48,803::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2098::DEBUG::2012-08-30 13:12:48,804::task::588::TaskManager.Task::(_updateState) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::moving from state init -> state preparing
>Thread-2098::INFO::2012-08-30 13:12:48,804::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2098::DEBUG::2012-08-30 13:12:48,804::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`77545583-7d2f-42e7-8ee9-c6295ab1055a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2098::DEBUG::2012-08-30 13:12:48,804::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2098::DEBUG::2012-08-30 13:12:48,805::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2098::DEBUG::2012-08-30 13:12:48,805::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`77545583-7d2f-42e7-8ee9-c6295ab1055a`::Granted request
>Thread-2098::DEBUG::2012-08-30 13:12:48,805::task::817::TaskManager.Task::(resourceAcquired) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2098::DEBUG::2012-08-30 13:12:48,805::task::978::TaskManager.Task::(_decref) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::ref 1 aborting False
>Thread-2098::INFO::2012-08-30 13:12:48,808::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2098::DEBUG::2012-08-30 13:12:48,809::task::1172::TaskManager.Task::(prepare) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2098::DEBUG::2012-08-30 13:12:48,809::task::588::TaskManager.Task::(_updateState) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::moving from state preparing -> state finished
>Thread-2098::DEBUG::2012-08-30 13:12:48,809::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2098::DEBUG::2012-08-30 13:12:48,809::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2098::DEBUG::2012-08-30 13:12:48,810::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2098::DEBUG::2012-08-30 13:12:48,810::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2098::DEBUG::2012-08-30 13:12:48,810::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2098::DEBUG::2012-08-30 13:12:48,810::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2098::DEBUG::2012-08-30 13:12:48,811::task::978::TaskManager.Task::(_decref) Task=`ed5c7eb5-f1db-479d-8a63-63d35e47f513`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:12:50,752::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:12:53,622::task::588::TaskManager.Task::(_updateState) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:12:53,622::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:12:53,623::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`71070b69-0bad-4e0c-98f5-2857f246babe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:12:53,623::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:12:53,624::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:12:53,624::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`71070b69-0bad-4e0c-98f5-2857f246babe`::Granted request
>Thread-40::DEBUG::2012-08-30 13:12:53,624::task::817::TaskManager.Task::(resourceAcquired) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:12:53,624::task::978::TaskManager.Task::(_decref) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:12:53,626::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:12:53,628::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:12:53,629::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:12:53,630::task::1172::TaskManager.Task::(prepare) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:12:53,630::task::588::TaskManager.Task::(_updateState) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:12:53,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:12:53,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:12:53,631::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:12:53,631::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:12:53,631::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:12:53,631::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:12:53,631::task::978::TaskManager.Task::(_decref) Task=`fd8445c3-7a25-448f-92e5-e509262e684d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:12:53,632::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:53,632::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:53,633::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:12:53,633::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:12:55,753::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:12:56,179::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:12:58,634::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:00,754::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:00,754::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2102::DEBUG::2012-08-30 13:13:02,019::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2102::DEBUG::2012-08-30 13:13:02,019::task::588::TaskManager.Task::(_updateState) Task=`8d37e533-12c7-495c-b064-ac5d01837169`::moving from state init -> state preparing
>Thread-2102::INFO::2012-08-30 13:13:02,019::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2102::INFO::2012-08-30 13:13:02,020::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2102::DEBUG::2012-08-30 13:13:02,020::task::1172::TaskManager.Task::(prepare) Task=`8d37e533-12c7-495c-b064-ac5d01837169`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2102::DEBUG::2012-08-30 13:13:02,020::task::588::TaskManager.Task::(_updateState) Task=`8d37e533-12c7-495c-b064-ac5d01837169`::moving from state preparing -> state finished
>Thread-2102::DEBUG::2012-08-30 13:13:02,020::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2102::DEBUG::2012-08-30 13:13:02,021::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2102::DEBUG::2012-08-30 13:13:02,021::task::978::TaskManager.Task::(_decref) Task=`8d37e533-12c7-495c-b064-ac5d01837169`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:13:03,185::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:13:03,636::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2103::DEBUG::2012-08-30 13:13:05,178::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2104::DEBUG::2012-08-30 13:13:05,179::task::588::TaskManager.Task::(_updateState) Task=`d827489b-d98c-4e9e-8b85-32babf3a2f56`::moving from state init -> state preparing
>Thread-2103::DEBUG::2012-08-30 13:13:05,179::task::588::TaskManager.Task::(_updateState) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::moving from state init -> state preparing
>Thread-2104::INFO::2012-08-30 13:13:05,179::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2103::INFO::2012-08-30 13:13:05,179::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2104::INFO::2012-08-30 13:13:05,180::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177693367004', 'lastCheck': 1346303581.0220251, 'code': 0, 'valid': True}}
>Thread-2103::DEBUG::2012-08-30 13:13:05,180::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0c4c7978-f91d-4c86-ac29-9ef6f1de79d2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2104::DEBUG::2012-08-30 13:13:05,180::task::1172::TaskManager.Task::(prepare) Task=`d827489b-d98c-4e9e-8b85-32babf3a2f56`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00177693367004', 'lastCheck': 1346303581.0220251, 'code': 0, 'valid': True}}
>Thread-2103::DEBUG::2012-08-30 13:13:05,180::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2104::DEBUG::2012-08-30 13:13:05,181::task::588::TaskManager.Task::(_updateState) Task=`d827489b-d98c-4e9e-8b85-32babf3a2f56`::moving from state preparing -> state finished
>Thread-2103::DEBUG::2012-08-30 13:13:05,181::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2104::DEBUG::2012-08-30 13:13:05,181::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2103::DEBUG::2012-08-30 13:13:05,181::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0c4c7978-f91d-4c86-ac29-9ef6f1de79d2`::Granted request
>Thread-2104::DEBUG::2012-08-30 13:13:05,181::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2103::DEBUG::2012-08-30 13:13:05,182::task::817::TaskManager.Task::(resourceAcquired) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2104::DEBUG::2012-08-30 13:13:05,182::task::978::TaskManager.Task::(_decref) Task=`d827489b-d98c-4e9e-8b85-32babf3a2f56`::ref 0 aborting False
>Thread-2103::DEBUG::2012-08-30 13:13:05,182::task::978::TaskManager.Task::(_decref) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::ref 1 aborting False
>Thread-2103::INFO::2012-08-30 13:13:05,187::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2103::DEBUG::2012-08-30 13:13:05,187::task::1172::TaskManager.Task::(prepare) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2103::DEBUG::2012-08-30 13:13:05,187::task::588::TaskManager.Task::(_updateState) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::moving from state preparing -> state finished
>Thread-2103::DEBUG::2012-08-30 13:13:05,188::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2103::DEBUG::2012-08-30 13:13:05,188::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2103::DEBUG::2012-08-30 13:13:05,188::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2103::DEBUG::2012-08-30 13:13:05,188::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2103::DEBUG::2012-08-30 13:13:05,189::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2103::DEBUG::2012-08-30 13:13:05,189::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2103::DEBUG::2012-08-30 13:13:05,189::task::978::TaskManager.Task::(_decref) Task=`4c27295d-0005-4375-a23a-5e853d578bda`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:13:05,755::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2105::DEBUG::2012-08-30 13:13:08,363::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2105::DEBUG::2012-08-30 13:13:08,363::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:13:08,637::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:08,637::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:10,756::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:13,638::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:15,757::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:15,758::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:18,640::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2107::DEBUG::2012-08-30 13:13:19,442::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2107::DEBUG::2012-08-30 13:13:19,443::task::588::TaskManager.Task::(_updateState) Task=`d2c8ae02-5d20-4ca7-83af-9c8803dfcaf6`::moving from state init -> state preparing
>Thread-2107::INFO::2012-08-30 13:13:19,443::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2107::INFO::2012-08-30 13:13:19,444::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2107::DEBUG::2012-08-30 13:13:19,444::task::1172::TaskManager.Task::(prepare) Task=`d2c8ae02-5d20-4ca7-83af-9c8803dfcaf6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2107::DEBUG::2012-08-30 13:13:19,444::task::588::TaskManager.Task::(_updateState) Task=`d2c8ae02-5d20-4ca7-83af-9c8803dfcaf6`::moving from state preparing -> state finished
>Thread-2107::DEBUG::2012-08-30 13:13:19,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2107::DEBUG::2012-08-30 13:13:19,445::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2107::DEBUG::2012-08-30 13:13:19,445::task::978::TaskManager.Task::(_decref) Task=`d2c8ae02-5d20-4ca7-83af-9c8803dfcaf6`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:13:20,758::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:23,641::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:23,642::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2109::DEBUG::2012-08-30 13:13:24,181::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2109::DEBUG::2012-08-30 13:13:24,182::task::588::TaskManager.Task::(_updateState) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::moving from state init -> state preparing
>Thread-2109::INFO::2012-08-30 13:13:24,182::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2109::DEBUG::2012-08-30 13:13:24,182::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`de5fdbfd-cea7-4751-ac24-ac975612cfe6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2109::DEBUG::2012-08-30 13:13:24,182::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2109::DEBUG::2012-08-30 13:13:24,183::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2109::DEBUG::2012-08-30 13:13:24,183::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`de5fdbfd-cea7-4751-ac24-ac975612cfe6`::Granted request
>Thread-2109::DEBUG::2012-08-30 13:13:24,183::task::817::TaskManager.Task::(resourceAcquired) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2109::DEBUG::2012-08-30 13:13:24,184::task::978::TaskManager.Task::(_decref) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::ref 1 aborting False
>Thread-2109::INFO::2012-08-30 13:13:24,186::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2109::DEBUG::2012-08-30 13:13:24,187::task::1172::TaskManager.Task::(prepare) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2109::DEBUG::2012-08-30 13:13:24,187::task::588::TaskManager.Task::(_updateState) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::moving from state preparing -> state finished
>Thread-2109::DEBUG::2012-08-30 13:13:24,187::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2109::DEBUG::2012-08-30 13:13:24,187::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2109::DEBUG::2012-08-30 13:13:24,188::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2109::DEBUG::2012-08-30 13:13:24,188::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2109::DEBUG::2012-08-30 13:13:24,188::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2109::DEBUG::2012-08-30 13:13:24,188::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2109::DEBUG::2012-08-30 13:13:24,189::task::978::TaskManager.Task::(_decref) Task=`a9654a7b-db3b-407a-a1b1-22bf00aee87a`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:13:25,759::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:13:26,204::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:13:28,643::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:13:30,760::task::588::TaskManager.Task::(_updateState) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:13:30,761::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:13:30,761::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`24f2d2cb-a228-49fc-89f6-b9da26b1d806`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:13:30,761::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:13:30,762::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:13:30,762::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`24f2d2cb-a228-49fc-89f6-b9da26b1d806`::Granted request
>Thread-104::DEBUG::2012-08-30 13:13:30,762::task::817::TaskManager.Task::(resourceAcquired) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:13:30,762::task::978::TaskManager.Task::(_decref) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:13:30,764::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:13:30,766::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:13:30,767::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:13:30,767::task::1172::TaskManager.Task::(prepare) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:13:30,768::task::588::TaskManager.Task::(_updateState) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:13:30,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:13:30,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:13:30,768::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:13:30,769::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:13:30,769::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:13:30,769::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:13:30,769::task::978::TaskManager.Task::(_decref) Task=`6aa20735-1437-47a5-89dd-d1d7cd4d914b`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:13:30,770::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:30,770::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:30,771::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:30,771::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:13:33,211::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:13:33,645::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:35,772::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:38,646::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:38,646::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2112::DEBUG::2012-08-30 13:13:39,538::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2112::DEBUG::2012-08-30 13:13:39,539::task::588::TaskManager.Task::(_updateState) Task=`d64154fa-d839-4575-a519-ff5151f6689e`::moving from state init -> state preparing
>Thread-2112::INFO::2012-08-30 13:13:39,539::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2112::INFO::2012-08-30 13:13:39,539::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2112::DEBUG::2012-08-30 13:13:39,540::task::1172::TaskManager.Task::(prepare) Task=`d64154fa-d839-4575-a519-ff5151f6689e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2112::DEBUG::2012-08-30 13:13:39,540::task::588::TaskManager.Task::(_updateState) Task=`d64154fa-d839-4575-a519-ff5151f6689e`::moving from state preparing -> state finished
>Thread-2112::DEBUG::2012-08-30 13:13:39,540::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2112::DEBUG::2012-08-30 13:13:39,540::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2112::DEBUG::2012-08-30 13:13:39,540::task::978::TaskManager.Task::(_decref) Task=`d64154fa-d839-4575-a519-ff5151f6689e`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:13:40,773::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2113::DEBUG::2012-08-30 13:13:42,397::task::588::TaskManager.Task::(_updateState) Task=`c4c60f6f-754f-4185-a930-85c5d308ca67`::moving from state init -> state preparing
>Thread-2113::INFO::2012-08-30 13:13:42,397::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2113::INFO::2012-08-30 13:13:42,397::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016348361969', 'lastCheck': 1346303621.047157, 'code': 0, 'valid': True}}
>Thread-2113::DEBUG::2012-08-30 13:13:42,397::task::1172::TaskManager.Task::(prepare) Task=`c4c60f6f-754f-4185-a930-85c5d308ca67`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.0016348361969', 'lastCheck': 1346303621.047157, 'code': 0, 'valid': True}}
>Thread-2113::DEBUG::2012-08-30 13:13:42,398::task::588::TaskManager.Task::(_updateState) Task=`c4c60f6f-754f-4185-a930-85c5d308ca67`::moving from state preparing -> state finished
>Thread-2113::DEBUG::2012-08-30 13:13:42,398::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2113::DEBUG::2012-08-30 13:13:42,398::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2113::DEBUG::2012-08-30 13:13:42,398::task::978::TaskManager.Task::(_decref) Task=`c4c60f6f-754f-4185-a930-85c5d308ca67`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:13:43,647::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2114::DEBUG::2012-08-30 13:13:45,344::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2114::DEBUG::2012-08-30 13:13:45,345::task::588::TaskManager.Task::(_updateState) Task=`bfd955b7-024d-4862-b982-69069277160b`::moving from state init -> state preparing
>Thread-2114::INFO::2012-08-30 13:13:45,345::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2114::DEBUG::2012-08-30 13:13:45,346::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1e24f4ce-df10-4a7b-9516-8fb374427447`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2114::DEBUG::2012-08-30 13:13:45,346::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2114::DEBUG::2012-08-30 13:13:45,346::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2114::DEBUG::2012-08-30 13:13:45,346::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1e24f4ce-df10-4a7b-9516-8fb374427447`::Granted request
>Thread-2114::DEBUG::2012-08-30 13:13:45,347::task::817::TaskManager.Task::(resourceAcquired) Task=`bfd955b7-024d-4862-b982-69069277160b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2114::DEBUG::2012-08-30 13:13:45,347::task::978::TaskManager.Task::(_decref) Task=`bfd955b7-024d-4862-b982-69069277160b`::ref 1 aborting False
>Thread-2114::INFO::2012-08-30 13:13:45,350::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2114::DEBUG::2012-08-30 13:13:45,351::task::1172::TaskManager.Task::(prepare) Task=`bfd955b7-024d-4862-b982-69069277160b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2114::DEBUG::2012-08-30 13:13:45,351::task::588::TaskManager.Task::(_updateState) Task=`bfd955b7-024d-4862-b982-69069277160b`::moving from state preparing -> state finished
>Thread-2114::DEBUG::2012-08-30 13:13:45,351::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2114::DEBUG::2012-08-30 13:13:45,351::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2114::DEBUG::2012-08-30 13:13:45,352::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2114::DEBUG::2012-08-30 13:13:45,352::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2114::DEBUG::2012-08-30 13:13:45,352::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2114::DEBUG::2012-08-30 13:13:45,352::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2114::DEBUG::2012-08-30 13:13:45,352::task::978::TaskManager.Task::(_decref) Task=`bfd955b7-024d-4862-b982-69069277160b`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:13:45,774::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:45,774::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2115::DEBUG::2012-08-30 13:13:48,048::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2115::DEBUG::2012-08-30 13:13:48,048::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:13:48,649::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:50,775::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:13:53,650::task::588::TaskManager.Task::(_updateState) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:13:53,650::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:13:53,651::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f8c0b1d3-788e-43b8-bb1b-345205f4fd28`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:13:53,651::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:13:53,651::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:13:53,651::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`f8c0b1d3-788e-43b8-bb1b-345205f4fd28`::Granted request
>Thread-40::DEBUG::2012-08-30 13:13:53,652::task::817::TaskManager.Task::(resourceAcquired) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:13:53,652::task::978::TaskManager.Task::(_decref) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:13:53,654::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:13:53,656::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:13:53,657::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:13:53,657::task::1172::TaskManager.Task::(prepare) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:13:53,657::task::588::TaskManager.Task::(_updateState) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:13:53,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:13:53,658::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:13:53,658::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:13:53,658::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:13:53,659::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:13:53,659::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:13:53,659::task::978::TaskManager.Task::(_decref) Task=`c76465fe-23bc-4004-8102-0a1d9a652d89`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:13:53,659::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:53,660::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:53,660::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:13:53,661::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:13:55,776::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:13:56,231::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:13:58,662::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:00,777::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:00,777::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2117::DEBUG::2012-08-30 13:14:01,492::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2117::DEBUG::2012-08-30 13:14:01,492::task::588::TaskManager.Task::(_updateState) Task=`b38c5db0-2d26-49f2-b427-d564fcac11f3`::moving from state init -> state preparing
>Thread-2117::INFO::2012-08-30 13:14:01,492::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2117::INFO::2012-08-30 13:14:01,493::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2117::DEBUG::2012-08-30 13:14:01,493::task::1172::TaskManager.Task::(prepare) Task=`b38c5db0-2d26-49f2-b427-d564fcac11f3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2117::DEBUG::2012-08-30 13:14:01,493::task::588::TaskManager.Task::(_updateState) Task=`b38c5db0-2d26-49f2-b427-d564fcac11f3`::moving from state preparing -> state finished
>Thread-2117::DEBUG::2012-08-30 13:14:01,493::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2117::DEBUG::2012-08-30 13:14:01,494::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2117::DEBUG::2012-08-30 13:14:01,494::task::978::TaskManager.Task::(_decref) Task=`b38c5db0-2d26-49f2-b427-d564fcac11f3`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:14:03,238::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:14:03,663::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:05,778::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2119::DEBUG::2012-08-30 13:14:08,208::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2119::DEBUG::2012-08-30 13:14:08,208::task::588::TaskManager.Task::(_updateState) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::moving from state init -> state preparing
>Thread-2119::INFO::2012-08-30 13:14:08,208::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2119::DEBUG::2012-08-30 13:14:08,209::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`aac84ae2-41d5-4787-9e2a-6c44ed7e4789`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2119::DEBUG::2012-08-30 13:14:08,209::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2119::DEBUG::2012-08-30 13:14:08,209::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2119::DEBUG::2012-08-30 13:14:08,209::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`aac84ae2-41d5-4787-9e2a-6c44ed7e4789`::Granted request
>Thread-2119::DEBUG::2012-08-30 13:14:08,210::task::817::TaskManager.Task::(resourceAcquired) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2119::DEBUG::2012-08-30 13:14:08,210::task::978::TaskManager.Task::(_decref) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::ref 1 aborting False
>Thread-2119::INFO::2012-08-30 13:14:08,213::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2119::DEBUG::2012-08-30 13:14:08,213::task::1172::TaskManager.Task::(prepare) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2119::DEBUG::2012-08-30 13:14:08,213::task::588::TaskManager.Task::(_updateState) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::moving from state preparing -> state finished
>Thread-2119::DEBUG::2012-08-30 13:14:08,214::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2119::DEBUG::2012-08-30 13:14:08,214::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2119::DEBUG::2012-08-30 13:14:08,214::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2119::DEBUG::2012-08-30 13:14:08,214::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2119::DEBUG::2012-08-30 13:14:08,215::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2119::DEBUG::2012-08-30 13:14:08,215::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2119::DEBUG::2012-08-30 13:14:08,215::task::978::TaskManager.Task::(_decref) Task=`2c1196fc-5b7b-44b1-90e2-15dc76647aff`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:14:08,664::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:08,665::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:10,779::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:13,666::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:15,780::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:15,781::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:18,667::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:20,782::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2122::DEBUG::2012-08-30 13:14:23,045::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2122::DEBUG::2012-08-30 13:14:23,046::task::588::TaskManager.Task::(_updateState) Task=`480e2b5c-287e-462b-8e05-4c22f964b896`::moving from state init -> state preparing
>Thread-2122::INFO::2012-08-30 13:14:23,046::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2122::INFO::2012-08-30 13:14:23,047::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2122::DEBUG::2012-08-30 13:14:23,047::task::1172::TaskManager.Task::(prepare) Task=`480e2b5c-287e-462b-8e05-4c22f964b896`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2122::DEBUG::2012-08-30 13:14:23,047::task::588::TaskManager.Task::(_updateState) Task=`480e2b5c-287e-462b-8e05-4c22f964b896`::moving from state preparing -> state finished
>Thread-2122::DEBUG::2012-08-30 13:14:23,047::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2122::DEBUG::2012-08-30 13:14:23,047::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2122::DEBUG::2012-08-30 13:14:23,048::task::978::TaskManager.Task::(_decref) Task=`480e2b5c-287e-462b-8e05-4c22f964b896`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:14:23,669::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:23,669::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2123::DEBUG::2012-08-30 13:14:25,528::task::588::TaskManager.Task::(_updateState) Task=`2c42f780-a9da-4662-a20b-350217ddf2b4`::moving from state init -> state preparing
>Thread-2123::INFO::2012-08-30 13:14:25,528::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2123::INFO::2012-08-30 13:14:25,528::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163984298706', 'lastCheck': 1346303661.074084, 'code': 0, 'valid': True}}
>Thread-2123::DEBUG::2012-08-30 13:14:25,528::task::1172::TaskManager.Task::(prepare) Task=`2c42f780-a9da-4662-a20b-350217ddf2b4`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163984298706', 'lastCheck': 1346303661.074084, 'code': 0, 'valid': True}}
>Thread-2123::DEBUG::2012-08-30 13:14:25,529::task::588::TaskManager.Task::(_updateState) Task=`2c42f780-a9da-4662-a20b-350217ddf2b4`::moving from state preparing -> state finished
>Thread-2123::DEBUG::2012-08-30 13:14:25,529::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2123::DEBUG::2012-08-30 13:14:25,529::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2123::DEBUG::2012-08-30 13:14:25,529::task::978::TaskManager.Task::(_decref) Task=`2c42f780-a9da-4662-a20b-350217ddf2b4`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:14:25,783::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:14:26,257::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2124::DEBUG::2012-08-30 13:14:27,131::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2124::DEBUG::2012-08-30 13:14:27,132::task::588::TaskManager.Task::(_updateState) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::moving from state init -> state preparing
>Thread-2124::INFO::2012-08-30 13:14:27,132::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2124::DEBUG::2012-08-30 13:14:27,132::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ecc57c27-4fe5-448b-8632-6b389bf765e9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2124::DEBUG::2012-08-30 13:14:27,132::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2124::DEBUG::2012-08-30 13:14:27,133::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2124::DEBUG::2012-08-30 13:14:27,133::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ecc57c27-4fe5-448b-8632-6b389bf765e9`::Granted request
>Thread-2124::DEBUG::2012-08-30 13:14:27,133::task::817::TaskManager.Task::(resourceAcquired) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2124::DEBUG::2012-08-30 13:14:27,134::task::978::TaskManager.Task::(_decref) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::ref 1 aborting False
>Thread-2124::INFO::2012-08-30 13:14:27,137::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2124::DEBUG::2012-08-30 13:14:27,137::task::1172::TaskManager.Task::(prepare) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2124::DEBUG::2012-08-30 13:14:27,137::task::588::TaskManager.Task::(_updateState) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::moving from state preparing -> state finished
>Thread-2124::DEBUG::2012-08-30 13:14:27,137::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2124::DEBUG::2012-08-30 13:14:27,137::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2124::DEBUG::2012-08-30 13:14:27,138::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2124::DEBUG::2012-08-30 13:14:27,138::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2124::DEBUG::2012-08-30 13:14:27,138::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2124::DEBUG::2012-08-30 13:14:27,138::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2124::DEBUG::2012-08-30 13:14:27,139::task::978::TaskManager.Task::(_decref) Task=`8c7e0910-3fc7-47ac-8403-11a015b0ed58`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:14:28,671::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2125::DEBUG::2012-08-30 13:14:28,929::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2125::DEBUG::2012-08-30 13:14:28,930::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::DEBUG::2012-08-30 13:14:30,784::task::588::TaskManager.Task::(_updateState) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:14:30,784::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:14:30,784::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`eae0f68f-8cf3-435e-a18e-c21caa227db2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:14:30,784::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:14:30,785::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:14:30,785::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`eae0f68f-8cf3-435e-a18e-c21caa227db2`::Granted request
>Thread-104::DEBUG::2012-08-30 13:14:30,785::task::817::TaskManager.Task::(resourceAcquired) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:14:30,786::task::978::TaskManager.Task::(_decref) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:14:30,787::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:14:30,789::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:14:30,790::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:14:30,791::task::1172::TaskManager.Task::(prepare) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:14:30,791::task::588::TaskManager.Task::(_updateState) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:14:30,791::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:14:30,791::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:14:30,791::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:14:30,792::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:14:30,792::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:14:30,792::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:14:30,792::task::978::TaskManager.Task::(_decref) Task=`7905bdf9-6b77-4253-964b-b5aae81ce9fa`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:14:30,793::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:30,793::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:30,794::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:30,794::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:14:33,264::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:14:33,672::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:35,795::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:38,673::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:38,674::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2128::DEBUG::2012-08-30 13:14:40,568::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2128::DEBUG::2012-08-30 13:14:40,568::task::588::TaskManager.Task::(_updateState) Task=`b50612e2-c1c8-423b-ad5b-54c67dc0cf46`::moving from state init -> state preparing
>Thread-2128::INFO::2012-08-30 13:14:40,568::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2128::INFO::2012-08-30 13:14:40,569::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2128::DEBUG::2012-08-30 13:14:40,569::task::1172::TaskManager.Task::(prepare) Task=`b50612e2-c1c8-423b-ad5b-54c67dc0cf46`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2128::DEBUG::2012-08-30 13:14:40,569::task::588::TaskManager.Task::(_updateState) Task=`b50612e2-c1c8-423b-ad5b-54c67dc0cf46`::moving from state preparing -> state finished
>Thread-2128::DEBUG::2012-08-30 13:14:40,569::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2128::DEBUG::2012-08-30 13:14:40,570::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2128::DEBUG::2012-08-30 13:14:40,570::task::978::TaskManager.Task::(_decref) Task=`b50612e2-c1c8-423b-ad5b-54c67dc0cf46`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:14:40,796::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:43,675::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2129::DEBUG::2012-08-30 13:14:44,426::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2129::DEBUG::2012-08-30 13:14:44,426::task::588::TaskManager.Task::(_updateState) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::moving from state init -> state preparing
>Thread-2129::INFO::2012-08-30 13:14:44,426::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2129::DEBUG::2012-08-30 13:14:44,427::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`db98f15b-e79e-4ffe-99ab-dd1a44e4a99e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2129::DEBUG::2012-08-30 13:14:44,427::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2129::DEBUG::2012-08-30 13:14:44,427::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2129::DEBUG::2012-08-30 13:14:44,427::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`db98f15b-e79e-4ffe-99ab-dd1a44e4a99e`::Granted request
>Thread-2129::DEBUG::2012-08-30 13:14:44,428::task::817::TaskManager.Task::(resourceAcquired) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2129::DEBUG::2012-08-30 13:14:44,428::task::978::TaskManager.Task::(_decref) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::ref 1 aborting False
>Thread-2129::INFO::2012-08-30 13:14:44,431::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2129::DEBUG::2012-08-30 13:14:44,431::task::1172::TaskManager.Task::(prepare) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2129::DEBUG::2012-08-30 13:14:44,431::task::588::TaskManager.Task::(_updateState) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::moving from state preparing -> state finished
>Thread-2129::DEBUG::2012-08-30 13:14:44,432::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2129::DEBUG::2012-08-30 13:14:44,432::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2129::DEBUG::2012-08-30 13:14:44,432::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2129::DEBUG::2012-08-30 13:14:44,432::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2129::DEBUG::2012-08-30 13:14:44,433::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2129::DEBUG::2012-08-30 13:14:44,433::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2129::DEBUG::2012-08-30 13:14:44,433::task::978::TaskManager.Task::(_decref) Task=`1103a833-31d1-47c2-a61a-3365a2989ec7`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:14:45,797::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:45,797::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:48,676::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:50,798::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:14:53,677::task::588::TaskManager.Task::(_updateState) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:14:53,678::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:14:53,678::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`47150741-1c2c-4ec2-a42a-3db704fb7d48`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:14:53,678::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:14:53,679::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:14:53,679::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`47150741-1c2c-4ec2-a42a-3db704fb7d48`::Granted request
>Thread-40::DEBUG::2012-08-30 13:14:53,679::task::817::TaskManager.Task::(resourceAcquired) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:14:53,680::task::978::TaskManager.Task::(_decref) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:14:53,681::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:14:53,683::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:14:53,684::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:14:53,685::task::1172::TaskManager.Task::(prepare) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:14:53,685::task::588::TaskManager.Task::(_updateState) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:14:53,685::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:14:53,685::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:14:53,686::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:14:53,686::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:14:53,686::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:14:53,686::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:14:53,687::task::978::TaskManager.Task::(_decref) Task=`ce5d17d1-308e-4cf8-8a4f-f4189bd84451`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:14:53,687::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:53,687::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:53,688::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:14:53,688::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:14:55,799::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:14:56,284::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:14:58,690::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2132::DEBUG::2012-08-30 13:15:00,011::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2132::DEBUG::2012-08-30 13:15:00,011::task::588::TaskManager.Task::(_updateState) Task=`5497a39f-1491-436f-9c60-3e0699cb40b4`::moving from state init -> state preparing
>Thread-2132::INFO::2012-08-30 13:15:00,012::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2132::INFO::2012-08-30 13:15:00,012::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2132::DEBUG::2012-08-30 13:15:00,012::task::1172::TaskManager.Task::(prepare) Task=`5497a39f-1491-436f-9c60-3e0699cb40b4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2132::DEBUG::2012-08-30 13:15:00,012::task::588::TaskManager.Task::(_updateState) Task=`5497a39f-1491-436f-9c60-3e0699cb40b4`::moving from state preparing -> state finished
>Thread-2132::DEBUG::2012-08-30 13:15:00,013::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2132::DEBUG::2012-08-30 13:15:00,013::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2132::DEBUG::2012-08-30 13:15:00,013::task::978::TaskManager.Task::(_decref) Task=`5497a39f-1491-436f-9c60-3e0699cb40b4`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:15:00,800::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:00,801::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2133::DEBUG::2012-08-30 13:15:03,077::task::588::TaskManager.Task::(_updateState) Task=`ee3b83ef-cd03-47db-992b-b47ff26d4c71`::moving from state init -> state preparing
>Thread-2133::INFO::2012-08-30 13:15:03,077::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2133::INFO::2012-08-30 13:15:03,077::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00179195404053', 'lastCheck': 1346303701.100708, 'code': 0, 'valid': True}}
>Thread-2133::DEBUG::2012-08-30 13:15:03,078::task::1172::TaskManager.Task::(prepare) Task=`ee3b83ef-cd03-47db-992b-b47ff26d4c71`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00179195404053', 'lastCheck': 1346303701.100708, 'code': 0, 'valid': True}}
>Thread-2133::DEBUG::2012-08-30 13:15:03,078::task::588::TaskManager.Task::(_updateState) Task=`ee3b83ef-cd03-47db-992b-b47ff26d4c71`::moving from state preparing -> state finished
>Thread-2133::DEBUG::2012-08-30 13:15:03,078::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2133::DEBUG::2012-08-30 13:15:03,078::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2133::DEBUG::2012-08-30 13:15:03,078::task::978::TaskManager.Task::(_decref) Task=`ee3b83ef-cd03-47db-992b-b47ff26d4c71`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:15:03,290::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:15:03,691::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:05,801::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2134::DEBUG::2012-08-30 13:15:06,321::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2134::DEBUG::2012-08-30 13:15:06,321::task::588::TaskManager.Task::(_updateState) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::moving from state init -> state preparing
>Thread-2134::INFO::2012-08-30 13:15:06,321::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2134::DEBUG::2012-08-30 13:15:06,322::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a9fc6001-ad1b-4f00-b35c-9a9ff34f984b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2134::DEBUG::2012-08-30 13:15:06,322::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2134::DEBUG::2012-08-30 13:15:06,322::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2134::DEBUG::2012-08-30 13:15:06,322::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a9fc6001-ad1b-4f00-b35c-9a9ff34f984b`::Granted request
>Thread-2134::DEBUG::2012-08-30 13:15:06,323::task::817::TaskManager.Task::(resourceAcquired) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2134::DEBUG::2012-08-30 13:15:06,323::task::978::TaskManager.Task::(_decref) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::ref 1 aborting False
>Thread-2134::INFO::2012-08-30 13:15:06,326::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2134::DEBUG::2012-08-30 13:15:06,326::task::1172::TaskManager.Task::(prepare) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2134::DEBUG::2012-08-30 13:15:06,326::task::588::TaskManager.Task::(_updateState) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::moving from state preparing -> state finished
>Thread-2134::DEBUG::2012-08-30 13:15:06,327::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2134::DEBUG::2012-08-30 13:15:06,327::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2134::DEBUG::2012-08-30 13:15:06,327::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2134::DEBUG::2012-08-30 13:15:06,327::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2134::DEBUG::2012-08-30 13:15:06,328::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2134::DEBUG::2012-08-30 13:15:06,328::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2134::DEBUG::2012-08-30 13:15:06,328::task::978::TaskManager.Task::(_decref) Task=`f19341a9-0ef9-4360-87d2-4015637b8eda`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:15:08,692::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:08,692::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2135::DEBUG::2012-08-30 13:15:09,500::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2135::DEBUG::2012-08-30 13:15:09,501::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:15:10,802::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:13,693::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:15,803::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:15,804::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:18,695::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:20,805::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2137::DEBUG::2012-08-30 13:15:22,116::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2137::DEBUG::2012-08-30 13:15:22,117::task::588::TaskManager.Task::(_updateState) Task=`28a02c4e-b0ae-40a0-815a-85e572874024`::moving from state init -> state preparing
>Thread-2137::INFO::2012-08-30 13:15:22,117::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2137::INFO::2012-08-30 13:15:22,117::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2137::DEBUG::2012-08-30 13:15:22,117::task::1172::TaskManager.Task::(prepare) Task=`28a02c4e-b0ae-40a0-815a-85e572874024`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2137::DEBUG::2012-08-30 13:15:22,118::task::588::TaskManager.Task::(_updateState) Task=`28a02c4e-b0ae-40a0-815a-85e572874024`::moving from state preparing -> state finished
>Thread-2137::DEBUG::2012-08-30 13:15:22,118::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2137::DEBUG::2012-08-30 13:15:22,118::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2137::DEBUG::2012-08-30 13:15:22,118::task::978::TaskManager.Task::(_decref) Task=`28a02c4e-b0ae-40a0-815a-85e572874024`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:15:23,696::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:23,697::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:25,806::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:15:26,309::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2139::DEBUG::2012-08-30 13:15:28,339::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2139::DEBUG::2012-08-30 13:15:28,340::task::588::TaskManager.Task::(_updateState) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::moving from state init -> state preparing
>Thread-2139::INFO::2012-08-30 13:15:28,340::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2139::DEBUG::2012-08-30 13:15:28,340::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0724596-2461-4844-a836-23678d44f294`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2139::DEBUG::2012-08-30 13:15:28,341::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2139::DEBUG::2012-08-30 13:15:28,341::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2139::DEBUG::2012-08-30 13:15:28,341::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0724596-2461-4844-a836-23678d44f294`::Granted request
>Thread-2139::DEBUG::2012-08-30 13:15:28,342::task::817::TaskManager.Task::(resourceAcquired) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2139::DEBUG::2012-08-30 13:15:28,342::task::978::TaskManager.Task::(_decref) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::ref 1 aborting False
>Thread-2139::INFO::2012-08-30 13:15:28,345::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2139::DEBUG::2012-08-30 13:15:28,345::task::1172::TaskManager.Task::(prepare) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2139::DEBUG::2012-08-30 13:15:28,345::task::588::TaskManager.Task::(_updateState) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::moving from state preparing -> state finished
>Thread-2139::DEBUG::2012-08-30 13:15:28,345::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2139::DEBUG::2012-08-30 13:15:28,346::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2139::DEBUG::2012-08-30 13:15:28,346::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2139::DEBUG::2012-08-30 13:15:28,346::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2139::DEBUG::2012-08-30 13:15:28,346::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2139::DEBUG::2012-08-30 13:15:28,347::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2139::DEBUG::2012-08-30 13:15:28,347::task::978::TaskManager.Task::(_decref) Task=`4d9b5911-e95e-4943-b73c-3db4d1c0ff9d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:15:28,698::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:15:30,807::task::588::TaskManager.Task::(_updateState) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:15:30,807::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:15:30,808::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c168573c-0160-409e-8bac-5e6c60efc047`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:15:30,808::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:15:30,808::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:15:30,808::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c168573c-0160-409e-8bac-5e6c60efc047`::Granted request
>Thread-104::DEBUG::2012-08-30 13:15:30,809::task::817::TaskManager.Task::(resourceAcquired) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:15:30,809::task::978::TaskManager.Task::(_decref) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:15:30,810::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:15:30,813::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:15:30,814::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:15:30,814::task::1172::TaskManager.Task::(prepare) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:15:30,814::task::588::TaskManager.Task::(_updateState) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:15:30,814::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:15:30,815::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:15:30,815::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:15:30,815::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:15:30,815::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:15:30,816::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:15:30,816::task::978::TaskManager.Task::(_decref) Task=`edc2215f-0655-4950-a127-4fa09a43fcd6`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:15:30,816::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:30,817::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:30,817::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:30,817::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-26::DEBUG::2012-08-30 13:15:31,113::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:15:31,113::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-26::DEBUG::2012-08-30 13:15:31,114::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None)
>Thread-26::DEBUG::2012-08-30 13:15:31,222::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = '  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5
>Thread-26::WARNING::2012-08-30 13:15:31,223::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found']
>Thread-26::DEBUG::2012-08-30 13:15:31,223::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-26::DEBUG::2012-08-30 13:15:31,228::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:15:31,229::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend
>Thread-26::DEBUG::2012-08-30 13:15:31,231::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d']
>Thread-26::DEBUG::2012-08-30 13:15:31,232::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-26::WARNING::2012-08-30 13:15:31,233::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered
>Thread-26::WARNING::2012-08-30 13:15:31,233::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered
>VM Channels Listener::DEBUG::2012-08-30 13:15:33,317::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:15:33,700::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:35,818::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:38,701::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:38,702::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:40,819::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2147::DEBUG::2012-08-30 13:15:43,119::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2147::DEBUG::2012-08-30 13:15:43,119::task::588::TaskManager.Task::(_updateState) Task=`0d397ba6-4b4b-4496-944b-af34443256b2`::moving from state init -> state preparing
>Thread-2147::INFO::2012-08-30 13:15:43,119::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2147::INFO::2012-08-30 13:15:43,120::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2147::DEBUG::2012-08-30 13:15:43,120::task::1172::TaskManager.Task::(prepare) Task=`0d397ba6-4b4b-4496-944b-af34443256b2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2147::DEBUG::2012-08-30 13:15:43,120::task::588::TaskManager.Task::(_updateState) Task=`0d397ba6-4b4b-4496-944b-af34443256b2`::moving from state preparing -> state finished
>Thread-2147::DEBUG::2012-08-30 13:15:43,121::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2147::DEBUG::2012-08-30 13:15:43,121::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2147::DEBUG::2012-08-30 13:15:43,121::task::978::TaskManager.Task::(_decref) Task=`0d397ba6-4b4b-4496-944b-af34443256b2`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:15:43,703::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2148::DEBUG::2012-08-30 13:15:45,097::task::588::TaskManager.Task::(_updateState) Task=`8193a844-30c0-40a8-af49-f45f1675d169`::moving from state init -> state preparing
>Thread-2148::INFO::2012-08-30 13:15:45,098::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2148::INFO::2012-08-30 13:15:45,098::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163698196411', 'lastCheck': 1346303741.2457759, 'code': 0, 'valid': True}}
>Thread-2148::DEBUG::2012-08-30 13:15:45,098::task::1172::TaskManager.Task::(prepare) Task=`8193a844-30c0-40a8-af49-f45f1675d169`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163698196411', 'lastCheck': 1346303741.2457759, 'code': 0, 'valid': True}}
>Thread-2148::DEBUG::2012-08-30 13:15:45,098::task::588::TaskManager.Task::(_updateState) Task=`8193a844-30c0-40a8-af49-f45f1675d169`::moving from state preparing -> state finished
>Thread-2148::DEBUG::2012-08-30 13:15:45,099::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2148::DEBUG::2012-08-30 13:15:45,099::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2148::DEBUG::2012-08-30 13:15:45,099::task::978::TaskManager.Task::(_decref) Task=`8193a844-30c0-40a8-af49-f45f1675d169`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:15:45,820::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:45,821::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2149::DEBUG::2012-08-30 13:15:47,201::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2149::DEBUG::2012-08-30 13:15:47,202::task::588::TaskManager.Task::(_updateState) Task=`76963a33-87ef-4caa-9455-616dd605360a`::moving from state init -> state preparing
>Thread-2149::INFO::2012-08-30 13:15:47,202::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2149::DEBUG::2012-08-30 13:15:47,202::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`53d8b6d8-1e05-4696-9c07-1a825bc44616`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2149::DEBUG::2012-08-30 13:15:47,203::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2149::DEBUG::2012-08-30 13:15:47,203::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2149::DEBUG::2012-08-30 13:15:47,203::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`53d8b6d8-1e05-4696-9c07-1a825bc44616`::Granted request
>Thread-2149::DEBUG::2012-08-30 13:15:47,204::task::817::TaskManager.Task::(resourceAcquired) Task=`76963a33-87ef-4caa-9455-616dd605360a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2149::DEBUG::2012-08-30 13:15:47,204::task::978::TaskManager.Task::(_decref) Task=`76963a33-87ef-4caa-9455-616dd605360a`::ref 1 aborting False
>Thread-2149::INFO::2012-08-30 13:15:47,207::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2149::DEBUG::2012-08-30 13:15:47,207::task::1172::TaskManager.Task::(prepare) Task=`76963a33-87ef-4caa-9455-616dd605360a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2149::DEBUG::2012-08-30 13:15:47,207::task::588::TaskManager.Task::(_updateState) Task=`76963a33-87ef-4caa-9455-616dd605360a`::moving from state preparing -> state finished
>Thread-2149::DEBUG::2012-08-30 13:15:47,207::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2149::DEBUG::2012-08-30 13:15:47,208::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2149::DEBUG::2012-08-30 13:15:47,208::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2149::DEBUG::2012-08-30 13:15:47,208::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2149::DEBUG::2012-08-30 13:15:47,208::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2149::DEBUG::2012-08-30 13:15:47,209::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2149::DEBUG::2012-08-30 13:15:47,209::task::978::TaskManager.Task::(_decref) Task=`76963a33-87ef-4caa-9455-616dd605360a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:15:48,704::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2150::DEBUG::2012-08-30 13:15:49,099::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2150::DEBUG::2012-08-30 13:15:49,099::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:15:50,822::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:15:53,705::task::588::TaskManager.Task::(_updateState) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:15:53,706::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:15:53,706::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e5bc4a05-b145-4a3a-89bd-39f424f0ddb3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:15:53,706::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:15:53,707::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:15:53,707::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`e5bc4a05-b145-4a3a-89bd-39f424f0ddb3`::Granted request
>Thread-40::DEBUG::2012-08-30 13:15:53,707::task::817::TaskManager.Task::(resourceAcquired) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:15:53,708::task::978::TaskManager.Task::(_decref) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:15:53,709::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:15:53,711::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:15:53,713::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:15:53,713::task::1172::TaskManager.Task::(prepare) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:15:53,713::task::588::TaskManager.Task::(_updateState) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:15:53,713::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:15:53,713::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:15:53,714::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:15:53,714::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:15:53,714::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:15:53,714::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:15:53,715::task::978::TaskManager.Task::(_decref) Task=`d920e025-6f09-4849-b82b-b7d14c87c98d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:15:53,715::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:53,716::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:53,716::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:15:53,716::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:15:55,823::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:15:56,337::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:15:58,718::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:00,824::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:00,824::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2152::DEBUG::2012-08-30 13:16:01,190::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2152::DEBUG::2012-08-30 13:16:01,191::task::588::TaskManager.Task::(_updateState) Task=`ac7e0c36-b5c3-4e19-934f-001cd5f0b7c5`::moving from state init -> state preparing
>Thread-2152::INFO::2012-08-30 13:16:01,191::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2152::INFO::2012-08-30 13:16:01,192::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2152::DEBUG::2012-08-30 13:16:01,192::task::1172::TaskManager.Task::(prepare) Task=`ac7e0c36-b5c3-4e19-934f-001cd5f0b7c5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2152::DEBUG::2012-08-30 13:16:01,192::task::588::TaskManager.Task::(_updateState) Task=`ac7e0c36-b5c3-4e19-934f-001cd5f0b7c5`::moving from state preparing -> state finished
>Thread-2152::DEBUG::2012-08-30 13:16:01,192::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2152::DEBUG::2012-08-30 13:16:01,193::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2152::DEBUG::2012-08-30 13:16:01,193::task::978::TaskManager.Task::(_decref) Task=`ac7e0c36-b5c3-4e19-934f-001cd5f0b7c5`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:16:03,343::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:16:03,719::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2154::DEBUG::2012-08-30 13:16:05,082::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2154::DEBUG::2012-08-30 13:16:05,082::task::588::TaskManager.Task::(_updateState) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::moving from state init -> state preparing
>Thread-2154::INFO::2012-08-30 13:16:05,082::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2154::DEBUG::2012-08-30 13:16:05,083::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f28ca09c-02e2-4518-91e8-54ce54ad2312`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2154::DEBUG::2012-08-30 13:16:05,083::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2154::DEBUG::2012-08-30 13:16:05,083::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2154::DEBUG::2012-08-30 13:16:05,083::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f28ca09c-02e2-4518-91e8-54ce54ad2312`::Granted request
>Thread-2154::DEBUG::2012-08-30 13:16:05,084::task::817::TaskManager.Task::(resourceAcquired) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2154::DEBUG::2012-08-30 13:16:05,084::task::978::TaskManager.Task::(_decref) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::ref 1 aborting False
>Thread-2154::INFO::2012-08-30 13:16:05,087::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2154::DEBUG::2012-08-30 13:16:05,087::task::1172::TaskManager.Task::(prepare) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2154::DEBUG::2012-08-30 13:16:05,087::task::588::TaskManager.Task::(_updateState) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::moving from state preparing -> state finished
>Thread-2154::DEBUG::2012-08-30 13:16:05,088::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2154::DEBUG::2012-08-30 13:16:05,088::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2154::DEBUG::2012-08-30 13:16:05,088::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2154::DEBUG::2012-08-30 13:16:05,088::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2154::DEBUG::2012-08-30 13:16:05,089::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2154::DEBUG::2012-08-30 13:16:05,089::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2154::DEBUG::2012-08-30 13:16:05,089::task::978::TaskManager.Task::(_decref) Task=`213b1b72-b753-4dfc-a95a-3cde85e1a1e0`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:16:05,825::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:08,720::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:08,721::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:10,826::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:13,722::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:15,827::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:15,827::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2157::DEBUG::2012-08-30 13:16:17,448::task::588::TaskManager.Task::(_updateState) Task=`81bee071-6cb5-4bfb-8edb-50ed12ffdffd`::moving from state init -> state preparing
>Thread-2157::INFO::2012-08-30 13:16:17,448::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2157::INFO::2012-08-30 13:16:17,448::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346303771.2661681, 'code': 0, 'valid': True}}
>Thread-2157::DEBUG::2012-08-30 13:16:17,449::task::1172::TaskManager.Task::(prepare) Task=`81bee071-6cb5-4bfb-8edb-50ed12ffdffd`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346303771.2661681, 'code': 0, 'valid': True}}
>Thread-2157::DEBUG::2012-08-30 13:16:17,449::task::588::TaskManager.Task::(_updateState) Task=`81bee071-6cb5-4bfb-8edb-50ed12ffdffd`::moving from state preparing -> state finished
>Thread-2157::DEBUG::2012-08-30 13:16:17,449::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2157::DEBUG::2012-08-30 13:16:17,449::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2157::DEBUG::2012-08-30 13:16:17,450::task::978::TaskManager.Task::(_decref) Task=`81bee071-6cb5-4bfb-8edb-50ed12ffdffd`::ref 0 aborting False
>Thread-2158::DEBUG::2012-08-30 13:16:18,404::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2158::DEBUG::2012-08-30 13:16:18,405::task::588::TaskManager.Task::(_updateState) Task=`067b857b-384c-49a3-877f-12c21788f436`::moving from state init -> state preparing
>Thread-2158::INFO::2012-08-30 13:16:18,405::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2158::INFO::2012-08-30 13:16:18,405::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2158::DEBUG::2012-08-30 13:16:18,406::task::1172::TaskManager.Task::(prepare) Task=`067b857b-384c-49a3-877f-12c21788f436`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2158::DEBUG::2012-08-30 13:16:18,406::task::588::TaskManager.Task::(_updateState) Task=`067b857b-384c-49a3-877f-12c21788f436`::moving from state preparing -> state finished
>Thread-2158::DEBUG::2012-08-30 13:16:18,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2158::DEBUG::2012-08-30 13:16:18,406::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2158::DEBUG::2012-08-30 13:16:18,407::task::978::TaskManager.Task::(_decref) Task=`067b857b-384c-49a3-877f-12c21788f436`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:16:18,723::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2159::DEBUG::2012-08-30 13:16:20,586::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2159::DEBUG::2012-08-30 13:16:20,587::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:16:20,828::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2160::DEBUG::2012-08-30 13:16:23,583::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2160::DEBUG::2012-08-30 13:16:23,584::task::588::TaskManager.Task::(_updateState) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::moving from state init -> state preparing
>Thread-2160::INFO::2012-08-30 13:16:23,584::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2160::DEBUG::2012-08-30 13:16:23,584::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2cd7321d-4daf-4df0-be90-5f6128d0c560`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2160::DEBUG::2012-08-30 13:16:23,585::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2160::DEBUG::2012-08-30 13:16:23,585::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2160::DEBUG::2012-08-30 13:16:23,585::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2cd7321d-4daf-4df0-be90-5f6128d0c560`::Granted request
>Thread-2160::DEBUG::2012-08-30 13:16:23,585::task::817::TaskManager.Task::(resourceAcquired) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2160::DEBUG::2012-08-30 13:16:23,586::task::978::TaskManager.Task::(_decref) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::ref 1 aborting False
>Thread-2160::INFO::2012-08-30 13:16:23,589::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2160::DEBUG::2012-08-30 13:16:23,589::task::1172::TaskManager.Task::(prepare) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2160::DEBUG::2012-08-30 13:16:23,589::task::588::TaskManager.Task::(_updateState) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::moving from state preparing -> state finished
>Thread-2160::DEBUG::2012-08-30 13:16:23,589::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2160::DEBUG::2012-08-30 13:16:23,590::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2160::DEBUG::2012-08-30 13:16:23,590::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2160::DEBUG::2012-08-30 13:16:23,590::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2160::DEBUG::2012-08-30 13:16:23,590::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2160::DEBUG::2012-08-30 13:16:23,591::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2160::DEBUG::2012-08-30 13:16:23,591::task::978::TaskManager.Task::(_decref) Task=`8a6da6c4-2eb2-4105-9f00-1b7f3b13fa3c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:16:23,724::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:23,725::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:25,829::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:16:26,362::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:16:28,726::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:16:30,830::task::588::TaskManager.Task::(_updateState) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:16:30,830::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:16:30,831::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`269ab286-3a0f-45e1-93f4-184f8a7b808c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:16:30,831::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:16:30,831::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:16:30,831::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`269ab286-3a0f-45e1-93f4-184f8a7b808c`::Granted request
>Thread-104::DEBUG::2012-08-30 13:16:30,832::task::817::TaskManager.Task::(resourceAcquired) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:16:30,832::task::978::TaskManager.Task::(_decref) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:16:30,834::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:16:30,836::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:16:30,837::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:16:30,837::task::1172::TaskManager.Task::(prepare) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:16:30,837::task::588::TaskManager.Task::(_updateState) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:16:30,838::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:16:30,838::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:16:30,838::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:16:30,838::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:16:30,839::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:16:30,839::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:16:30,839::task::978::TaskManager.Task::(_decref) Task=`424a4ccd-d51f-4f8e-aa49-200168c8a4c2`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:16:30,839::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:30,840::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:30,840::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:30,841::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:16:33,366::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:16:33,727::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:35,842::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2163::DEBUG::2012-08-30 13:16:37,708::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2163::DEBUG::2012-08-30 13:16:37,708::task::588::TaskManager.Task::(_updateState) Task=`58a13bbc-4625-4e73-a8d5-72da1f8f0443`::moving from state init -> state preparing
>Thread-2163::INFO::2012-08-30 13:16:37,709::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2163::INFO::2012-08-30 13:16:37,709::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2163::DEBUG::2012-08-30 13:16:37,709::task::1172::TaskManager.Task::(prepare) Task=`58a13bbc-4625-4e73-a8d5-72da1f8f0443`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2163::DEBUG::2012-08-30 13:16:37,709::task::588::TaskManager.Task::(_updateState) Task=`58a13bbc-4625-4e73-a8d5-72da1f8f0443`::moving from state preparing -> state finished
>Thread-2163::DEBUG::2012-08-30 13:16:37,710::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2163::DEBUG::2012-08-30 13:16:37,710::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2163::DEBUG::2012-08-30 13:16:37,710::task::978::TaskManager.Task::(_decref) Task=`58a13bbc-4625-4e73-a8d5-72da1f8f0443`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:16:38,728::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:38,729::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:40,842::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2165::DEBUG::2012-08-30 13:16:42,494::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2165::DEBUG::2012-08-30 13:16:42,495::task::588::TaskManager.Task::(_updateState) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::moving from state init -> state preparing
>Thread-2165::INFO::2012-08-30 13:16:42,495::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2165::DEBUG::2012-08-30 13:16:42,496::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`87f1bfd4-e1af-4160-8261-0345e4124b7c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2165::DEBUG::2012-08-30 13:16:42,496::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2165::DEBUG::2012-08-30 13:16:42,496::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2165::DEBUG::2012-08-30 13:16:42,496::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`87f1bfd4-e1af-4160-8261-0345e4124b7c`::Granted request
>Thread-2165::DEBUG::2012-08-30 13:16:42,497::task::817::TaskManager.Task::(resourceAcquired) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2165::DEBUG::2012-08-30 13:16:42,497::task::978::TaskManager.Task::(_decref) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::ref 1 aborting False
>Thread-2165::INFO::2012-08-30 13:16:42,500::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2165::DEBUG::2012-08-30 13:16:42,500::task::1172::TaskManager.Task::(prepare) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2165::DEBUG::2012-08-30 13:16:42,500::task::588::TaskManager.Task::(_updateState) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::moving from state preparing -> state finished
>Thread-2165::DEBUG::2012-08-30 13:16:42,501::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2165::DEBUG::2012-08-30 13:16:42,501::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2165::DEBUG::2012-08-30 13:16:42,501::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2165::DEBUG::2012-08-30 13:16:42,501::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2165::DEBUG::2012-08-30 13:16:42,502::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2165::DEBUG::2012-08-30 13:16:42,502::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2165::DEBUG::2012-08-30 13:16:42,502::task::978::TaskManager.Task::(_decref) Task=`51f5eba3-e5a8-4116-81d6-8a9022bd2a8d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:16:43,730::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:45,843::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:45,844::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:48,731::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:16:50,845::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:16:53,732::task::588::TaskManager.Task::(_updateState) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:16:53,733::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:16:53,733::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`24b659ec-30a1-45e0-aba6-4eeddebc09d4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:16:53,734::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:16:53,734::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:16:53,734::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`24b659ec-30a1-45e0-aba6-4eeddebc09d4`::Granted request
>Thread-40::DEBUG::2012-08-30 13:16:53,735::task::817::TaskManager.Task::(resourceAcquired) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:16:53,735::task::978::TaskManager.Task::(_decref) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:16:53,737::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:16:53,739::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:16:53,740::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:16:53,740::task::1172::TaskManager.Task::(prepare) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:16:53,740::task::588::TaskManager.Task::(_updateState) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:16:53,740::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:16:53,741::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:16:53,741::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:16:53,741::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:16:53,741::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:16:53,742::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:16:53,742::task::978::TaskManager.Task::(_decref) Task=`b2be1e51-3aa7-472a-9176-233e1e1a1b7b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:16:53,742::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:53,743::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:53,743::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:16:53,744::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2167::DEBUG::2012-08-30 13:16:55,046::task::588::TaskManager.Task::(_updateState) Task=`6cd31a34-05c9-4b97-82fb-eb367510df25`::moving from state init -> state preparing
>Thread-2167::INFO::2012-08-30 13:16:55,046::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2167::INFO::2012-08-30 13:16:55,046::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178194046021', 'lastCheck': 1346303811.29179, 'code': 0, 'valid': True}}
>Thread-2167::DEBUG::2012-08-30 13:16:55,047::task::1172::TaskManager.Task::(prepare) Task=`6cd31a34-05c9-4b97-82fb-eb367510df25`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178194046021', 'lastCheck': 1346303811.29179, 'code': 0, 'valid': True}}
>Thread-2167::DEBUG::2012-08-30 13:16:55,047::task::588::TaskManager.Task::(_updateState) Task=`6cd31a34-05c9-4b97-82fb-eb367510df25`::moving from state preparing -> state finished
>Thread-2167::DEBUG::2012-08-30 13:16:55,047::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2167::DEBUG::2012-08-30 13:16:55,047::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2167::DEBUG::2012-08-30 13:16:55,048::task::978::TaskManager.Task::(_decref) Task=`6cd31a34-05c9-4b97-82fb-eb367510df25`::ref 0 aborting False
>Thread-2168::DEBUG::2012-08-30 13:16:55,049::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2168::DEBUG::2012-08-30 13:16:55,050::task::588::TaskManager.Task::(_updateState) Task=`9ca27cd1-f236-4d7d-a8b0-7ad32c82dd4e`::moving from state init -> state preparing
>Thread-2168::INFO::2012-08-30 13:16:55,050::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2168::INFO::2012-08-30 13:16:55,050::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2168::DEBUG::2012-08-30 13:16:55,051::task::1172::TaskManager.Task::(prepare) Task=`9ca27cd1-f236-4d7d-a8b0-7ad32c82dd4e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2168::DEBUG::2012-08-30 13:16:55,051::task::588::TaskManager.Task::(_updateState) Task=`9ca27cd1-f236-4d7d-a8b0-7ad32c82dd4e`::moving from state preparing -> state finished
>Thread-2168::DEBUG::2012-08-30 13:16:55,051::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2168::DEBUG::2012-08-30 13:16:55,051::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2168::DEBUG::2012-08-30 13:16:55,051::task::978::TaskManager.Task::(_decref) Task=`9ca27cd1-f236-4d7d-a8b0-7ad32c82dd4e`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:16:55,846::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:16:56,387::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2169::DEBUG::2012-08-30 13:16:58,191::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2170::DEBUG::2012-08-30 13:16:58,191::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2169::DEBUG::2012-08-30 13:16:58,192::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-2170::DEBUG::2012-08-30 13:16:58,192::task::588::TaskManager.Task::(_updateState) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::moving from state init -> state preparing
>Thread-2170::INFO::2012-08-30 13:16:58,193::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2170::DEBUG::2012-08-30 13:16:58,194::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c7ef120e-c4e4-41b7-9806-fa3dc35e2a21`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2170::DEBUG::2012-08-30 13:16:58,194::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2170::DEBUG::2012-08-30 13:16:58,194::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2170::DEBUG::2012-08-30 13:16:58,195::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c7ef120e-c4e4-41b7-9806-fa3dc35e2a21`::Granted request
>Thread-2170::DEBUG::2012-08-30 13:16:58,195::task::817::TaskManager.Task::(resourceAcquired) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2170::DEBUG::2012-08-30 13:16:58,195::task::978::TaskManager.Task::(_decref) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::ref 1 aborting False
>Thread-2170::INFO::2012-08-30 13:16:58,198::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2170::DEBUG::2012-08-30 13:16:58,198::task::1172::TaskManager.Task::(prepare) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2170::DEBUG::2012-08-30 13:16:58,199::task::588::TaskManager.Task::(_updateState) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::moving from state preparing -> state finished
>Thread-2170::DEBUG::2012-08-30 13:16:58,199::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2170::DEBUG::2012-08-30 13:16:58,199::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2170::DEBUG::2012-08-30 13:16:58,199::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2170::DEBUG::2012-08-30 13:16:58,200::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2170::DEBUG::2012-08-30 13:16:58,200::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2170::DEBUG::2012-08-30 13:16:58,200::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2170::DEBUG::2012-08-30 13:16:58,200::task::978::TaskManager.Task::(_decref) Task=`a06665e2-544c-4470-a4c9-f544d6fb1a9d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:16:58,745::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:00,847::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:00,847::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:17:03,393::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:17:03,746::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:05,848::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:08,747::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:08,748::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:10,849::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2173::DEBUG::2012-08-30 13:17:11,439::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2173::DEBUG::2012-08-30 13:17:11,439::task::588::TaskManager.Task::(_updateState) Task=`2638c9bb-7364-4483-8bf9-bfdbd2dddc43`::moving from state init -> state preparing
>Thread-2173::INFO::2012-08-30 13:17:11,440::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2173::INFO::2012-08-30 13:17:11,440::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2173::DEBUG::2012-08-30 13:17:11,440::task::1172::TaskManager.Task::(prepare) Task=`2638c9bb-7364-4483-8bf9-bfdbd2dddc43`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2173::DEBUG::2012-08-30 13:17:11,440::task::588::TaskManager.Task::(_updateState) Task=`2638c9bb-7364-4483-8bf9-bfdbd2dddc43`::moving from state preparing -> state finished
>Thread-2173::DEBUG::2012-08-30 13:17:11,440::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2173::DEBUG::2012-08-30 13:17:11,441::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2173::DEBUG::2012-08-30 13:17:11,441::task::978::TaskManager.Task::(_decref) Task=`2638c9bb-7364-4483-8bf9-bfdbd2dddc43`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:17:13,749::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2175::DEBUG::2012-08-30 13:17:14,662::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2175::DEBUG::2012-08-30 13:17:14,662::task::588::TaskManager.Task::(_updateState) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::moving from state init -> state preparing
>Thread-2175::INFO::2012-08-30 13:17:14,663::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2175::DEBUG::2012-08-30 13:17:14,663::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`139bf660-d5eb-4bb4-9f89-b603e0812e74`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2175::DEBUG::2012-08-30 13:17:14,663::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2175::DEBUG::2012-08-30 13:17:14,664::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2175::DEBUG::2012-08-30 13:17:14,664::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`139bf660-d5eb-4bb4-9f89-b603e0812e74`::Granted request
>Thread-2175::DEBUG::2012-08-30 13:17:14,664::task::817::TaskManager.Task::(resourceAcquired) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2175::DEBUG::2012-08-30 13:17:14,664::task::978::TaskManager.Task::(_decref) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::ref 1 aborting False
>Thread-2175::INFO::2012-08-30 13:17:14,667::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2175::DEBUG::2012-08-30 13:17:14,668::task::1172::TaskManager.Task::(prepare) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2175::DEBUG::2012-08-30 13:17:14,668::task::588::TaskManager.Task::(_updateState) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::moving from state preparing -> state finished
>Thread-2175::DEBUG::2012-08-30 13:17:14,668::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2175::DEBUG::2012-08-30 13:17:14,668::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2175::DEBUG::2012-08-30 13:17:14,669::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2175::DEBUG::2012-08-30 13:17:14,669::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2175::DEBUG::2012-08-30 13:17:14,669::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2175::DEBUG::2012-08-30 13:17:14,669::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2175::DEBUG::2012-08-30 13:17:14,670::task::978::TaskManager.Task::(_decref) Task=`6f9e0cb9-3dcf-4207-bf75-cf74268dd97b`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:17:15,850::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:15,851::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:18,750::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:20,851::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:23,752::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:23,752::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:25,852::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:17:26,411::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2177::DEBUG::2012-08-30 13:17:27,479::task::588::TaskManager.Task::(_updateState) Task=`de954292-7921-4f36-8b2d-30c84dcb05ec`::moving from state init -> state preparing
>Thread-2177::INFO::2012-08-30 13:17:27,479::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2177::INFO::2012-08-30 13:17:27,480::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303841.311954, 'code': 0, 'valid': True}}
>Thread-2177::DEBUG::2012-08-30 13:17:27,480::task::1172::TaskManager.Task::(prepare) Task=`de954292-7921-4f36-8b2d-30c84dcb05ec`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303841.311954, 'code': 0, 'valid': True}}
>Thread-2177::DEBUG::2012-08-30 13:17:27,480::task::588::TaskManager.Task::(_updateState) Task=`de954292-7921-4f36-8b2d-30c84dcb05ec`::moving from state preparing -> state finished
>Thread-2177::DEBUG::2012-08-30 13:17:27,480::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2177::DEBUG::2012-08-30 13:17:27,481::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2177::DEBUG::2012-08-30 13:17:27,481::task::978::TaskManager.Task::(_decref) Task=`de954292-7921-4f36-8b2d-30c84dcb05ec`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:17:28,754::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2178::DEBUG::2012-08-30 13:17:29,443::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2178::DEBUG::2012-08-30 13:17:29,443::task::588::TaskManager.Task::(_updateState) Task=`dce3bbc0-15c3-4d76-9c4f-b0c2d2cd77f7`::moving from state init -> state preparing
>Thread-2178::INFO::2012-08-30 13:17:29,443::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2178::INFO::2012-08-30 13:17:29,444::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2178::DEBUG::2012-08-30 13:17:29,444::task::1172::TaskManager.Task::(prepare) Task=`dce3bbc0-15c3-4d76-9c4f-b0c2d2cd77f7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2178::DEBUG::2012-08-30 13:17:29,444::task::588::TaskManager.Task::(_updateState) Task=`dce3bbc0-15c3-4d76-9c4f-b0c2d2cd77f7`::moving from state preparing -> state finished
>Thread-2178::DEBUG::2012-08-30 13:17:29,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2178::DEBUG::2012-08-30 13:17:29,445::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2178::DEBUG::2012-08-30 13:17:29,445::task::978::TaskManager.Task::(_decref) Task=`dce3bbc0-15c3-4d76-9c4f-b0c2d2cd77f7`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 13:17:30,853::task::588::TaskManager.Task::(_updateState) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:17:30,854::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:17:30,854::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`5e479691-3c05-415c-a345-977919d4a585`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:17:30,854::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:17:30,855::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:17:30,855::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`5e479691-3c05-415c-a345-977919d4a585`::Granted request
>Thread-104::DEBUG::2012-08-30 13:17:30,855::task::817::TaskManager.Task::(resourceAcquired) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:17:30,855::task::978::TaskManager.Task::(_decref) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:17:30,857::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:17:30,859::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:17:30,860::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:17:30,860::task::1172::TaskManager.Task::(prepare) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:17:30,861::task::588::TaskManager.Task::(_updateState) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:17:30,861::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:17:30,861::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:17:30,861::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:17:30,862::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:17:30,862::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:17:30,862::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:17:30,862::task::978::TaskManager.Task::(_decref) Task=`084c7cac-752f-482d-8138-39f1f717edbc`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:17:30,863::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:30,863::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:30,864::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:30,864::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2179::DEBUG::2012-08-30 13:17:31,092::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2179::DEBUG::2012-08-30 13:17:31,093::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-2180::DEBUG::2012-08-30 13:17:32,768::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2180::DEBUG::2012-08-30 13:17:32,768::task::588::TaskManager.Task::(_updateState) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::moving from state init -> state preparing
>Thread-2180::INFO::2012-08-30 13:17:32,768::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2180::DEBUG::2012-08-30 13:17:32,769::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`bff1fac9-1131-453e-b37c-f67087828d0e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2180::DEBUG::2012-08-30 13:17:32,769::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2180::DEBUG::2012-08-30 13:17:32,769::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2180::DEBUG::2012-08-30 13:17:32,770::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`bff1fac9-1131-453e-b37c-f67087828d0e`::Granted request
>Thread-2180::DEBUG::2012-08-30 13:17:32,770::task::817::TaskManager.Task::(resourceAcquired) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2180::DEBUG::2012-08-30 13:17:32,770::task::978::TaskManager.Task::(_decref) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::ref 1 aborting False
>Thread-2180::INFO::2012-08-30 13:17:32,773::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2180::DEBUG::2012-08-30 13:17:32,773::task::1172::TaskManager.Task::(prepare) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2180::DEBUG::2012-08-30 13:17:32,774::task::588::TaskManager.Task::(_updateState) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::moving from state preparing -> state finished
>Thread-2180::DEBUG::2012-08-30 13:17:32,774::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2180::DEBUG::2012-08-30 13:17:32,774::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2180::DEBUG::2012-08-30 13:17:32,774::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2180::DEBUG::2012-08-30 13:17:32,775::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2180::DEBUG::2012-08-30 13:17:32,775::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2180::DEBUG::2012-08-30 13:17:32,775::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2180::DEBUG::2012-08-30 13:17:32,775::task::978::TaskManager.Task::(_decref) Task=`e8b5d5b6-10df-40e0-9d04-b9150a7ba676`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:17:33,417::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:17:33,755::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:35,865::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:38,756::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:38,757::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:40,866::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:43,758::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2183::DEBUG::2012-08-30 13:17:45,493::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2183::DEBUG::2012-08-30 13:17:45,494::task::588::TaskManager.Task::(_updateState) Task=`e9ea8caa-5134-41be-a3d4-615cb75efd8c`::moving from state init -> state preparing
>Thread-2183::INFO::2012-08-30 13:17:45,494::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2183::INFO::2012-08-30 13:17:45,494::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2183::DEBUG::2012-08-30 13:17:45,494::task::1172::TaskManager.Task::(prepare) Task=`e9ea8caa-5134-41be-a3d4-615cb75efd8c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2183::DEBUG::2012-08-30 13:17:45,495::task::588::TaskManager.Task::(_updateState) Task=`e9ea8caa-5134-41be-a3d4-615cb75efd8c`::moving from state preparing -> state finished
>Thread-2183::DEBUG::2012-08-30 13:17:45,495::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2183::DEBUG::2012-08-30 13:17:45,495::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2183::DEBUG::2012-08-30 13:17:45,495::task::978::TaskManager.Task::(_decref) Task=`e9ea8caa-5134-41be-a3d4-615cb75efd8c`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:17:45,867::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:45,867::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2185::DEBUG::2012-08-30 13:17:48,714::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2185::DEBUG::2012-08-30 13:17:48,714::task::588::TaskManager.Task::(_updateState) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::moving from state init -> state preparing
>Thread-2185::INFO::2012-08-30 13:17:48,715::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2185::DEBUG::2012-08-30 13:17:48,715::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a78b4188-2e5c-4e23-bef6-c34002715659`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2185::DEBUG::2012-08-30 13:17:48,715::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2185::DEBUG::2012-08-30 13:17:48,716::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2185::DEBUG::2012-08-30 13:17:48,716::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`a78b4188-2e5c-4e23-bef6-c34002715659`::Granted request
>Thread-2185::DEBUG::2012-08-30 13:17:48,716::task::817::TaskManager.Task::(resourceAcquired) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2185::DEBUG::2012-08-30 13:17:48,716::task::978::TaskManager.Task::(_decref) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::ref 1 aborting False
>Thread-2185::INFO::2012-08-30 13:17:48,719::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2185::DEBUG::2012-08-30 13:17:48,720::task::1172::TaskManager.Task::(prepare) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2185::DEBUG::2012-08-30 13:17:48,720::task::588::TaskManager.Task::(_updateState) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::moving from state preparing -> state finished
>Thread-2185::DEBUG::2012-08-30 13:17:48,720::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2185::DEBUG::2012-08-30 13:17:48,720::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2185::DEBUG::2012-08-30 13:17:48,721::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2185::DEBUG::2012-08-30 13:17:48,721::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2185::DEBUG::2012-08-30 13:17:48,721::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2185::DEBUG::2012-08-30 13:17:48,721::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2185::DEBUG::2012-08-30 13:17:48,722::task::978::TaskManager.Task::(_decref) Task=`8a99c762-caef-49d9-b1d3-2b4f2f63deec`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:17:48,759::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:50,868::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:17:53,760::task::588::TaskManager.Task::(_updateState) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:17:53,760::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:17:53,761::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c8bcf974-c591-44fb-b238-84bd02425ea2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:17:53,761::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:17:53,762::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:17:53,762::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`c8bcf974-c591-44fb-b238-84bd02425ea2`::Granted request
>Thread-40::DEBUG::2012-08-30 13:17:53,762::task::817::TaskManager.Task::(resourceAcquired) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:17:53,762::task::978::TaskManager.Task::(_decref) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:17:53,764::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:17:53,766::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:17:53,767::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:17:53,768::task::1172::TaskManager.Task::(prepare) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:17:53,768::task::588::TaskManager.Task::(_updateState) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:17:53,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:17:53,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:17:53,769::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:17:53,769::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:17:53,769::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:17:53,769::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:17:53,769::task::978::TaskManager.Task::(_decref) Task=`8d5621e9-67c8-4e32-87bd-8153a694bf2d`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:17:53,770::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:53,770::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:53,771::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:17:53,771::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:17:55,869::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:17:56,432::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2187::DEBUG::2012-08-30 13:17:57,375::task::588::TaskManager.Task::(_updateState) Task=`09847a8e-5db9-4459-82fd-0d9d0bc9fb9e`::moving from state init -> state preparing
>Thread-2187::INFO::2012-08-30 13:17:57,375::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2187::INFO::2012-08-30 13:17:57,376::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178003311157', 'lastCheck': 1346303871.331373, 'code': 0, 'valid': True}}
>Thread-2187::DEBUG::2012-08-30 13:17:57,376::task::1172::TaskManager.Task::(prepare) Task=`09847a8e-5db9-4459-82fd-0d9d0bc9fb9e`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178003311157', 'lastCheck': 1346303871.331373, 'code': 0, 'valid': True}}
>Thread-2187::DEBUG::2012-08-30 13:17:57,376::task::588::TaskManager.Task::(_updateState) Task=`09847a8e-5db9-4459-82fd-0d9d0bc9fb9e`::moving from state preparing -> state finished
>Thread-2187::DEBUG::2012-08-30 13:17:57,376::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2187::DEBUG::2012-08-30 13:17:57,377::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2187::DEBUG::2012-08-30 13:17:57,377::task::978::TaskManager.Task::(_decref) Task=`09847a8e-5db9-4459-82fd-0d9d0bc9fb9e`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:17:58,772::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2188::DEBUG::2012-08-30 13:18:00,333::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2188::DEBUG::2012-08-30 13:18:00,333::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:18:00,870::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:00,870::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2189::DEBUG::2012-08-30 13:18:01,904::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2189::DEBUG::2012-08-30 13:18:01,904::task::588::TaskManager.Task::(_updateState) Task=`8394b7fb-ffeb-4f3c-b994-5a59d5e29daf`::moving from state init -> state preparing
>Thread-2189::INFO::2012-08-30 13:18:01,904::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2189::INFO::2012-08-30 13:18:01,905::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2189::DEBUG::2012-08-30 13:18:01,905::task::1172::TaskManager.Task::(prepare) Task=`8394b7fb-ffeb-4f3c-b994-5a59d5e29daf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2189::DEBUG::2012-08-30 13:18:01,905::task::588::TaskManager.Task::(_updateState) Task=`8394b7fb-ffeb-4f3c-b994-5a59d5e29daf`::moving from state preparing -> state finished
>Thread-2189::DEBUG::2012-08-30 13:18:01,905::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2189::DEBUG::2012-08-30 13:18:01,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2189::DEBUG::2012-08-30 13:18:01,906::task::978::TaskManager.Task::(_decref) Task=`8394b7fb-ffeb-4f3c-b994-5a59d5e29daf`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:18:03,439::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:18:03,774::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2190::DEBUG::2012-08-30 13:18:05,140::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2190::DEBUG::2012-08-30 13:18:05,141::task::588::TaskManager.Task::(_updateState) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::moving from state init -> state preparing
>Thread-2190::INFO::2012-08-30 13:18:05,142::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2190::DEBUG::2012-08-30 13:18:05,142::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`314ae09b-3a07-404e-9f3a-9c3f4a28cdb5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2190::DEBUG::2012-08-30 13:18:05,142::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2190::DEBUG::2012-08-30 13:18:05,142::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2190::DEBUG::2012-08-30 13:18:05,143::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`314ae09b-3a07-404e-9f3a-9c3f4a28cdb5`::Granted request
>Thread-2190::DEBUG::2012-08-30 13:18:05,143::task::817::TaskManager.Task::(resourceAcquired) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2190::DEBUG::2012-08-30 13:18:05,143::task::978::TaskManager.Task::(_decref) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::ref 1 aborting False
>Thread-2190::INFO::2012-08-30 13:18:05,146::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2190::DEBUG::2012-08-30 13:18:05,146::task::1172::TaskManager.Task::(prepare) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2190::DEBUG::2012-08-30 13:18:05,147::task::588::TaskManager.Task::(_updateState) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::moving from state preparing -> state finished
>Thread-2190::DEBUG::2012-08-30 13:18:05,147::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2190::DEBUG::2012-08-30 13:18:05,147::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2190::DEBUG::2012-08-30 13:18:05,147::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2190::DEBUG::2012-08-30 13:18:05,148::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2190::DEBUG::2012-08-30 13:18:05,148::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2190::DEBUG::2012-08-30 13:18:05,148::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2190::DEBUG::2012-08-30 13:18:05,148::task::978::TaskManager.Task::(_decref) Task=`be92767b-091d-4e9b-8fb0-588141ca273c`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:18:05,871::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:08,775::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:08,775::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:10,872::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:13,776::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:15,873::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:15,874::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2194::DEBUG::2012-08-30 13:18:18,330::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2194::DEBUG::2012-08-30 13:18:18,331::task::588::TaskManager.Task::(_updateState) Task=`6c3e817a-6470-4710-87b7-196d21f0b80a`::moving from state init -> state preparing
>Thread-2194::INFO::2012-08-30 13:18:18,331::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2194::INFO::2012-08-30 13:18:18,331::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2194::DEBUG::2012-08-30 13:18:18,332::task::1172::TaskManager.Task::(prepare) Task=`6c3e817a-6470-4710-87b7-196d21f0b80a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2194::DEBUG::2012-08-30 13:18:18,332::task::588::TaskManager.Task::(_updateState) Task=`6c3e817a-6470-4710-87b7-196d21f0b80a`::moving from state preparing -> state finished
>Thread-2194::DEBUG::2012-08-30 13:18:18,332::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2194::DEBUG::2012-08-30 13:18:18,332::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2194::DEBUG::2012-08-30 13:18:18,333::task::978::TaskManager.Task::(_decref) Task=`6c3e817a-6470-4710-87b7-196d21f0b80a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:18:18,778::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:20,874::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2196::DEBUG::2012-08-30 13:18:21,582::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2196::DEBUG::2012-08-30 13:18:21,582::task::588::TaskManager.Task::(_updateState) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::moving from state init -> state preparing
>Thread-2196::INFO::2012-08-30 13:18:21,583::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2196::DEBUG::2012-08-30 13:18:21,583::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0255acb-9f7a-4ae4-b5ef-51ac8c71899c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2196::DEBUG::2012-08-30 13:18:21,583::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2196::DEBUG::2012-08-30 13:18:21,583::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2196::DEBUG::2012-08-30 13:18:21,584::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d0255acb-9f7a-4ae4-b5ef-51ac8c71899c`::Granted request
>Thread-2196::DEBUG::2012-08-30 13:18:21,584::task::817::TaskManager.Task::(resourceAcquired) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2196::DEBUG::2012-08-30 13:18:21,584::task::978::TaskManager.Task::(_decref) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::ref 1 aborting False
>Thread-2196::INFO::2012-08-30 13:18:21,587::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2196::DEBUG::2012-08-30 13:18:21,587::task::1172::TaskManager.Task::(prepare) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2196::DEBUG::2012-08-30 13:18:21,588::task::588::TaskManager.Task::(_updateState) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::moving from state preparing -> state finished
>Thread-2196::DEBUG::2012-08-30 13:18:21,588::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2196::DEBUG::2012-08-30 13:18:21,588::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2196::DEBUG::2012-08-30 13:18:21,588::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2196::DEBUG::2012-08-30 13:18:21,589::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2196::DEBUG::2012-08-30 13:18:21,589::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2196::DEBUG::2012-08-30 13:18:21,589::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2196::DEBUG::2012-08-30 13:18:21,589::task::978::TaskManager.Task::(_decref) Task=`2aa5ab73-9334-4c1a-8dd8-b13d66524e2a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:18:23,779::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:23,780::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:25,875::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2197::DEBUG::2012-08-30 13:18:26,043::task::588::TaskManager.Task::(_updateState) Task=`a426d25a-7a15-40c8-ab77-fc251d0cf268`::moving from state init -> state preparing
>Thread-2197::INFO::2012-08-30 13:18:26,044::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2197::INFO::2012-08-30 13:18:26,044::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346303901.3513861, 'code': 0, 'valid': True}}
>Thread-2197::DEBUG::2012-08-30 13:18:26,044::task::1172::TaskManager.Task::(prepare) Task=`a426d25a-7a15-40c8-ab77-fc251d0cf268`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163602828979', 'lastCheck': 1346303901.3513861, 'code': 0, 'valid': True}}
>Thread-2197::DEBUG::2012-08-30 13:18:26,044::task::588::TaskManager.Task::(_updateState) Task=`a426d25a-7a15-40c8-ab77-fc251d0cf268`::moving from state preparing -> state finished
>Thread-2197::DEBUG::2012-08-30 13:18:26,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2197::DEBUG::2012-08-30 13:18:26,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2197::DEBUG::2012-08-30 13:18:26,045::task::978::TaskManager.Task::(_decref) Task=`a426d25a-7a15-40c8-ab77-fc251d0cf268`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:18:26,459::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:18:28,781::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2198::DEBUG::2012-08-30 13:18:29,192::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2198::DEBUG::2012-08-30 13:18:29,193::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::DEBUG::2012-08-30 13:18:30,876::task::588::TaskManager.Task::(_updateState) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:18:30,877::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:18:30,877::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`53ae24c0-0b3f-44bd-9387-f9b0f56acd48`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:18:30,877::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:18:30,878::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:18:30,878::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`53ae24c0-0b3f-44bd-9387-f9b0f56acd48`::Granted request
>Thread-104::DEBUG::2012-08-30 13:18:30,878::task::817::TaskManager.Task::(resourceAcquired) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:18:30,879::task::978::TaskManager.Task::(_decref) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:18:30,880::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:18:30,882::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:18:30,883::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:18:30,883::task::1172::TaskManager.Task::(prepare) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:18:30,884::task::588::TaskManager.Task::(_updateState) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:18:30,884::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:18:30,884::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:18:30,884::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:18:30,885::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:18:30,885::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:18:30,885::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:18:30,885::task::978::TaskManager.Task::(_decref) Task=`1b4bafcd-2ce6-4439-adb2-c8aeeabadeee`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:18:30,886::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:30,886::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:30,887::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:30,887::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:18:33,465::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:18:33,782::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2200::DEBUG::2012-08-30 13:18:34,442::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2200::DEBUG::2012-08-30 13:18:34,442::task::588::TaskManager.Task::(_updateState) Task=`9481aa17-b705-43f3-9a5d-388d693f327a`::moving from state init -> state preparing
>Thread-2200::INFO::2012-08-30 13:18:34,443::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2200::INFO::2012-08-30 13:18:34,443::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2200::DEBUG::2012-08-30 13:18:34,443::task::1172::TaskManager.Task::(prepare) Task=`9481aa17-b705-43f3-9a5d-388d693f327a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2200::DEBUG::2012-08-30 13:18:34,443::task::588::TaskManager.Task::(_updateState) Task=`9481aa17-b705-43f3-9a5d-388d693f327a`::moving from state preparing -> state finished
>Thread-2200::DEBUG::2012-08-30 13:18:34,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2200::DEBUG::2012-08-30 13:18:34,444::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2200::DEBUG::2012-08-30 13:18:34,444::task::978::TaskManager.Task::(_decref) Task=`9481aa17-b705-43f3-9a5d-388d693f327a`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:18:35,888::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2201::DEBUG::2012-08-30 13:18:37,616::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2201::DEBUG::2012-08-30 13:18:37,617::task::588::TaskManager.Task::(_updateState) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::moving from state init -> state preparing
>Thread-2201::INFO::2012-08-30 13:18:37,617::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2201::DEBUG::2012-08-30 13:18:37,617::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`10004e12-6aa8-4bac-a91d-3140fa3fdffe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2201::DEBUG::2012-08-30 13:18:37,617::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2201::DEBUG::2012-08-30 13:18:37,618::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2201::DEBUG::2012-08-30 13:18:37,618::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`10004e12-6aa8-4bac-a91d-3140fa3fdffe`::Granted request
>Thread-2201::DEBUG::2012-08-30 13:18:37,619::task::817::TaskManager.Task::(resourceAcquired) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2201::DEBUG::2012-08-30 13:18:37,619::task::978::TaskManager.Task::(_decref) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::ref 1 aborting False
>Thread-2201::INFO::2012-08-30 13:18:37,622::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2201::DEBUG::2012-08-30 13:18:37,622::task::1172::TaskManager.Task::(prepare) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2201::DEBUG::2012-08-30 13:18:37,622::task::588::TaskManager.Task::(_updateState) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::moving from state preparing -> state finished
>Thread-2201::DEBUG::2012-08-30 13:18:37,623::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2201::DEBUG::2012-08-30 13:18:37,623::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2201::DEBUG::2012-08-30 13:18:37,623::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2201::DEBUG::2012-08-30 13:18:37,623::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2201::DEBUG::2012-08-30 13:18:37,624::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2201::DEBUG::2012-08-30 13:18:37,624::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2201::DEBUG::2012-08-30 13:18:37,624::task::978::TaskManager.Task::(_decref) Task=`d17a8ba8-0f83-4c6b-8096-86af3bccec71`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:18:38,783::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:38,784::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:40,889::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:43,785::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:45,890::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:18:45,890::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:48,786::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2205::DEBUG::2012-08-30 13:18:50,807::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2205::DEBUG::2012-08-30 13:18:50,807::task::588::TaskManager.Task::(_updateState) Task=`afa4ea4a-0968-4d0a-97df-4bd41540d00e`::moving from state init -> state preparing
>Thread-2205::INFO::2012-08-30 13:18:50,808::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2205::INFO::2012-08-30 13:18:50,808::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2205::DEBUG::2012-08-30 13:18:50,808::task::1172::TaskManager.Task::(prepare) Task=`afa4ea4a-0968-4d0a-97df-4bd41540d00e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2205::DEBUG::2012-08-30 13:18:50,808::task::588::TaskManager.Task::(_updateState) Task=`afa4ea4a-0968-4d0a-97df-4bd41540d00e`::moving from state preparing -> state finished
>Thread-2205::DEBUG::2012-08-30 13:18:50,809::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2205::DEBUG::2012-08-30 13:18:50,809::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2205::DEBUG::2012-08-30 13:18:50,809::task::978::TaskManager.Task::(_decref) Task=`afa4ea4a-0968-4d0a-97df-4bd41540d00e`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:18:50,891::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:18:53,787::task::588::TaskManager.Task::(_updateState) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:18:53,788::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:18:53,789::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`2b0735cc-e720-41b9-9499-8c303f6bca00`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:18:53,789::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:18:53,789::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:18:53,789::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`2b0735cc-e720-41b9-9499-8c303f6bca00`::Granted request
>Thread-40::DEBUG::2012-08-30 13:18:53,790::task::817::TaskManager.Task::(resourceAcquired) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:18:53,790::task::978::TaskManager.Task::(_decref) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:18:53,792::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:18:53,794::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:18:53,795::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:18:53,795::task::1172::TaskManager.Task::(prepare) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:18:53,795::task::588::TaskManager.Task::(_updateState) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:18:53,796::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:18:53,797::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:18:53,797::task::978::TaskManager.Task::(_decref) Task=`eeeb859d-2d3e-48e1-b99b-eb1477d5b37c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:18:53,797::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:53,798::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:53,798::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:18:53,799::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2206::DEBUG::2012-08-30 13:18:54,215::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2207::DEBUG::2012-08-30 13:18:54,216::task::588::TaskManager.Task::(_updateState) Task=`c0a132de-5024-4f4e-9d61-3a0e7506d67c`::moving from state init -> state preparing
>Thread-2206::DEBUG::2012-08-30 13:18:54,216::task::588::TaskManager.Task::(_updateState) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::moving from state init -> state preparing
>Thread-2207::INFO::2012-08-30 13:18:54,216::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2206::INFO::2012-08-30 13:18:54,216::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2207::INFO::2012-08-30 13:18:54,217::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176405906677', 'lastCheck': 1346303931.3708031, 'code': 0, 'valid': True}}
>Thread-2206::DEBUG::2012-08-30 13:18:54,217::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0a7a9b3d-1544-415c-8e60-5ff616390283`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2207::DEBUG::2012-08-30 13:18:54,217::task::1172::TaskManager.Task::(prepare) Task=`c0a132de-5024-4f4e-9d61-3a0e7506d67c`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176405906677', 'lastCheck': 1346303931.3708031, 'code': 0, 'valid': True}}
>Thread-2206::DEBUG::2012-08-30 13:18:54,217::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2207::DEBUG::2012-08-30 13:18:54,218::task::588::TaskManager.Task::(_updateState) Task=`c0a132de-5024-4f4e-9d61-3a0e7506d67c`::moving from state preparing -> state finished
>Thread-2206::DEBUG::2012-08-30 13:18:54,218::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2207::DEBUG::2012-08-30 13:18:54,218::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2206::DEBUG::2012-08-30 13:18:54,218::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`0a7a9b3d-1544-415c-8e60-5ff616390283`::Granted request
>Thread-2207::DEBUG::2012-08-30 13:18:54,218::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2206::DEBUG::2012-08-30 13:18:54,219::task::817::TaskManager.Task::(resourceAcquired) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2207::DEBUG::2012-08-30 13:18:54,219::task::978::TaskManager.Task::(_decref) Task=`c0a132de-5024-4f4e-9d61-3a0e7506d67c`::ref 0 aborting False
>Thread-2206::DEBUG::2012-08-30 13:18:54,219::task::978::TaskManager.Task::(_decref) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::ref 1 aborting False
>Thread-2206::INFO::2012-08-30 13:18:54,224::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2206::DEBUG::2012-08-30 13:18:54,224::task::1172::TaskManager.Task::(prepare) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2206::DEBUG::2012-08-30 13:18:54,224::task::588::TaskManager.Task::(_updateState) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::moving from state preparing -> state finished
>Thread-2206::DEBUG::2012-08-30 13:18:54,225::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2206::DEBUG::2012-08-30 13:18:54,225::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2206::DEBUG::2012-08-30 13:18:54,225::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2206::DEBUG::2012-08-30 13:18:54,225::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2206::DEBUG::2012-08-30 13:18:54,226::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2206::DEBUG::2012-08-30 13:18:54,226::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2206::DEBUG::2012-08-30 13:18:54,226::task::978::TaskManager.Task::(_decref) Task=`db6dbe47-2f90-4fb0-9d59-8f5cb8ee16ec`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:18:55,892::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:18:56,486::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2208::DEBUG::2012-08-30 13:18:57,387::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2208::DEBUG::2012-08-30 13:18:57,387::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:18:58,800::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:00,893::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:00,894::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:19:03,493::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:19:03,801::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:05,895::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2210::DEBUG::2012-08-30 13:19:07,446::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2210::DEBUG::2012-08-30 13:19:07,446::task::588::TaskManager.Task::(_updateState) Task=`efe4b058-52f8-4045-bff1-4c28660c5ae7`::moving from state init -> state preparing
>Thread-2210::INFO::2012-08-30 13:19:07,447::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2210::INFO::2012-08-30 13:19:07,447::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2210::DEBUG::2012-08-30 13:19:07,447::task::1172::TaskManager.Task::(prepare) Task=`efe4b058-52f8-4045-bff1-4c28660c5ae7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2210::DEBUG::2012-08-30 13:19:07,447::task::588::TaskManager.Task::(_updateState) Task=`efe4b058-52f8-4045-bff1-4c28660c5ae7`::moving from state preparing -> state finished
>Thread-2210::DEBUG::2012-08-30 13:19:07,448::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2210::DEBUG::2012-08-30 13:19:07,448::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2210::DEBUG::2012-08-30 13:19:07,448::task::978::TaskManager.Task::(_decref) Task=`efe4b058-52f8-4045-bff1-4c28660c5ae7`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:19:08,802::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:08,803::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2212::DEBUG::2012-08-30 13:19:10,636::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2212::DEBUG::2012-08-30 13:19:10,636::task::588::TaskManager.Task::(_updateState) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::moving from state init -> state preparing
>Thread-2212::INFO::2012-08-30 13:19:10,636::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2212::DEBUG::2012-08-30 13:19:10,637::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e61e5291-17eb-4fb8-a881-fbae1d660ee6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2212::DEBUG::2012-08-30 13:19:10,637::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2212::DEBUG::2012-08-30 13:19:10,637::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2212::DEBUG::2012-08-30 13:19:10,637::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`e61e5291-17eb-4fb8-a881-fbae1d660ee6`::Granted request
>Thread-2212::DEBUG::2012-08-30 13:19:10,638::task::817::TaskManager.Task::(resourceAcquired) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2212::DEBUG::2012-08-30 13:19:10,638::task::978::TaskManager.Task::(_decref) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::ref 1 aborting False
>Thread-2212::INFO::2012-08-30 13:19:10,641::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2212::DEBUG::2012-08-30 13:19:10,641::task::1172::TaskManager.Task::(prepare) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2212::DEBUG::2012-08-30 13:19:10,641::task::588::TaskManager.Task::(_updateState) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::moving from state preparing -> state finished
>Thread-2212::DEBUG::2012-08-30 13:19:10,642::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2212::DEBUG::2012-08-30 13:19:10,642::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2212::DEBUG::2012-08-30 13:19:10,642::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2212::DEBUG::2012-08-30 13:19:10,642::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2212::DEBUG::2012-08-30 13:19:10,643::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2212::DEBUG::2012-08-30 13:19:10,643::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2212::DEBUG::2012-08-30 13:19:10,643::task::978::TaskManager.Task::(_decref) Task=`ff0788f6-a345-4953-bf68-d74cf06a668d`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:19:10,896::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:13,804::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:15,896::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:15,897::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:18,805::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:20,898::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2215::DEBUG::2012-08-30 13:19:22,410::task::588::TaskManager.Task::(_updateState) Task=`6420cfc3-02ab-47ed-be74-4fc0abdee7a5`::moving from state init -> state preparing
>Thread-2215::INFO::2012-08-30 13:19:22,410::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2215::INFO::2012-08-30 13:19:22,410::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303961.3905671, 'code': 0, 'valid': True}}
>Thread-2215::DEBUG::2012-08-30 13:19:22,410::task::1172::TaskManager.Task::(prepare) Task=`6420cfc3-02ab-47ed-be74-4fc0abdee7a5`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162696838379', 'lastCheck': 1346303961.3905671, 'code': 0, 'valid': True}}
>Thread-2215::DEBUG::2012-08-30 13:19:22,411::task::588::TaskManager.Task::(_updateState) Task=`6420cfc3-02ab-47ed-be74-4fc0abdee7a5`::moving from state preparing -> state finished
>Thread-2215::DEBUG::2012-08-30 13:19:22,411::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2215::DEBUG::2012-08-30 13:19:22,411::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2215::DEBUG::2012-08-30 13:19:22,411::task::978::TaskManager.Task::(_decref) Task=`6420cfc3-02ab-47ed-be74-4fc0abdee7a5`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:19:23,806::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:23,807::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2216::DEBUG::2012-08-30 13:19:23,970::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2216::DEBUG::2012-08-30 13:19:23,970::task::588::TaskManager.Task::(_updateState) Task=`aa12be73-2634-4bde-aeac-2e179fc9c524`::moving from state init -> state preparing
>Thread-2216::INFO::2012-08-30 13:19:23,971::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2216::INFO::2012-08-30 13:19:23,971::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2216::DEBUG::2012-08-30 13:19:23,971::task::1172::TaskManager.Task::(prepare) Task=`aa12be73-2634-4bde-aeac-2e179fc9c524`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2216::DEBUG::2012-08-30 13:19:23,971::task::588::TaskManager.Task::(_updateState) Task=`aa12be73-2634-4bde-aeac-2e179fc9c524`::moving from state preparing -> state finished
>Thread-2216::DEBUG::2012-08-30 13:19:23,972::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2216::DEBUG::2012-08-30 13:19:23,972::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2216::DEBUG::2012-08-30 13:19:23,972::task::978::TaskManager.Task::(_decref) Task=`aa12be73-2634-4bde-aeac-2e179fc9c524`::ref 0 aborting False
>Thread-2217::DEBUG::2012-08-30 13:19:25,539::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2217::DEBUG::2012-08-30 13:19:25,539::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:19:25,899::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:19:26,514::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2218::DEBUG::2012-08-30 13:19:27,147::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2218::DEBUG::2012-08-30 13:19:27,148::task::588::TaskManager.Task::(_updateState) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::moving from state init -> state preparing
>Thread-2218::INFO::2012-08-30 13:19:27,148::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2218::DEBUG::2012-08-30 13:19:27,148::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5d4603fb-8ae1-4822-9ece-992f1287559b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2218::DEBUG::2012-08-30 13:19:27,148::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2218::DEBUG::2012-08-30 13:19:27,149::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2218::DEBUG::2012-08-30 13:19:27,149::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5d4603fb-8ae1-4822-9ece-992f1287559b`::Granted request
>Thread-2218::DEBUG::2012-08-30 13:19:27,149::task::817::TaskManager.Task::(resourceAcquired) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2218::DEBUG::2012-08-30 13:19:27,149::task::978::TaskManager.Task::(_decref) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::ref 1 aborting False
>Thread-2218::INFO::2012-08-30 13:19:27,152::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2218::DEBUG::2012-08-30 13:19:27,153::task::1172::TaskManager.Task::(prepare) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2218::DEBUG::2012-08-30 13:19:27,153::task::588::TaskManager.Task::(_updateState) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::moving from state preparing -> state finished
>Thread-2218::DEBUG::2012-08-30 13:19:27,153::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2218::DEBUG::2012-08-30 13:19:27,153::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2218::DEBUG::2012-08-30 13:19:27,154::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2218::DEBUG::2012-08-30 13:19:27,154::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2218::DEBUG::2012-08-30 13:19:27,154::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2218::DEBUG::2012-08-30 13:19:27,154::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2218::DEBUG::2012-08-30 13:19:27,155::task::978::TaskManager.Task::(_decref) Task=`f25881d2-afe1-4188-84d2-f2b829cc7494`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:19:28,808::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:19:30,900::task::588::TaskManager.Task::(_updateState) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:19:30,900::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:19:30,900::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`dbc9cdb3-9e41-45fd-b44f-8de6b35d0ace`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:19:30,901::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:19:30,901::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:19:30,901::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`dbc9cdb3-9e41-45fd-b44f-8de6b35d0ace`::Granted request
>Thread-104::DEBUG::2012-08-30 13:19:30,901::task::817::TaskManager.Task::(resourceAcquired) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:19:30,902::task::978::TaskManager.Task::(_decref) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:19:30,903::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:19:30,905::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:19:30,906::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:19:30,907::task::1172::TaskManager.Task::(prepare) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:19:30,907::task::588::TaskManager.Task::(_updateState) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:19:30,907::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:19:30,907::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:19:30,908::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:19:30,908::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:19:30,908::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:19:30,908::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:19:30,909::task::978::TaskManager.Task::(_decref) Task=`61dad1bc-3e6a-4934-9653-908bab5d00cb`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:19:30,909::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:30,909::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:30,910::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:30,910::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:19:33,522::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:19:33,809::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:35,911::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:38,810::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:38,811::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2221::DEBUG::2012-08-30 13:19:40,425::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2221::DEBUG::2012-08-30 13:19:40,425::task::588::TaskManager.Task::(_updateState) Task=`3a91d453-2933-4caa-9f2f-811c1592aa1c`::moving from state init -> state preparing
>Thread-2221::INFO::2012-08-30 13:19:40,425::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2221::INFO::2012-08-30 13:19:40,426::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2221::DEBUG::2012-08-30 13:19:40,426::task::1172::TaskManager.Task::(prepare) Task=`3a91d453-2933-4caa-9f2f-811c1592aa1c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2221::DEBUG::2012-08-30 13:19:40,426::task::588::TaskManager.Task::(_updateState) Task=`3a91d453-2933-4caa-9f2f-811c1592aa1c`::moving from state preparing -> state finished
>Thread-2221::DEBUG::2012-08-30 13:19:40,426::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2221::DEBUG::2012-08-30 13:19:40,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2221::DEBUG::2012-08-30 13:19:40,427::task::978::TaskManager.Task::(_decref) Task=`3a91d453-2933-4caa-9f2f-811c1592aa1c`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:19:40,912::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2223::DEBUG::2012-08-30 13:19:43,585::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2223::DEBUG::2012-08-30 13:19:43,585::task::588::TaskManager.Task::(_updateState) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::moving from state init -> state preparing
>Thread-2223::INFO::2012-08-30 13:19:43,585::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2223::DEBUG::2012-08-30 13:19:43,586::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9aa7717c-9ff8-4353-b45f-2e0d21cd9a7b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2223::DEBUG::2012-08-30 13:19:43,586::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2223::DEBUG::2012-08-30 13:19:43,586::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2223::DEBUG::2012-08-30 13:19:43,586::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9aa7717c-9ff8-4353-b45f-2e0d21cd9a7b`::Granted request
>Thread-2223::DEBUG::2012-08-30 13:19:43,587::task::817::TaskManager.Task::(resourceAcquired) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2223::DEBUG::2012-08-30 13:19:43,587::task::978::TaskManager.Task::(_decref) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::ref 1 aborting False
>Thread-2223::INFO::2012-08-30 13:19:43,590::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2223::DEBUG::2012-08-30 13:19:43,590::task::1172::TaskManager.Task::(prepare) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2223::DEBUG::2012-08-30 13:19:43,590::task::588::TaskManager.Task::(_updateState) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::moving from state preparing -> state finished
>Thread-2223::DEBUG::2012-08-30 13:19:43,591::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2223::DEBUG::2012-08-30 13:19:43,591::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2223::DEBUG::2012-08-30 13:19:43,591::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2223::DEBUG::2012-08-30 13:19:43,591::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2223::DEBUG::2012-08-30 13:19:43,592::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2223::DEBUG::2012-08-30 13:19:43,592::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2223::DEBUG::2012-08-30 13:19:43,592::task::978::TaskManager.Task::(_decref) Task=`3001b9bc-e876-4e31-a46b-10dc4cc731b7`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:19:43,812::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:45,913::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:45,913::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:48,813::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:19:50,914::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2225::DEBUG::2012-08-30 13:19:51,503::task::588::TaskManager.Task::(_updateState) Task=`23c75423-035c-4585-8987-dd0cdae398cb`::moving from state init -> state preparing
>Thread-2225::INFO::2012-08-30 13:19:51,503::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2225::INFO::2012-08-30 13:19:51,503::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176191329956', 'lastCheck': 1346303991.4098279, 'code': 0, 'valid': True}}
>Thread-2225::DEBUG::2012-08-30 13:19:51,503::task::1172::TaskManager.Task::(prepare) Task=`23c75423-035c-4585-8987-dd0cdae398cb`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00176191329956', 'lastCheck': 1346303991.4098279, 'code': 0, 'valid': True}}
>Thread-2225::DEBUG::2012-08-30 13:19:51,504::task::588::TaskManager.Task::(_updateState) Task=`23c75423-035c-4585-8987-dd0cdae398cb`::moving from state preparing -> state finished
>Thread-2225::DEBUG::2012-08-30 13:19:51,504::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2225::DEBUG::2012-08-30 13:19:51,504::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2225::DEBUG::2012-08-30 13:19:51,504::task::978::TaskManager.Task::(_decref) Task=`23c75423-035c-4585-8987-dd0cdae398cb`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 13:19:53,814::task::588::TaskManager.Task::(_updateState) Task=`03e9a479-1232-442f-9b24-4653460e0153`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:19:53,815::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:19:53,815::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4ed66c77-f428-4c03-a363-4fd036a1d25a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:19:53,815::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:19:53,816::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:19:53,816::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4ed66c77-f428-4c03-a363-4fd036a1d25a`::Granted request
>Thread-40::DEBUG::2012-08-30 13:19:53,816::task::817::TaskManager.Task::(resourceAcquired) Task=`03e9a479-1232-442f-9b24-4653460e0153`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:19:53,816::task::978::TaskManager.Task::(_decref) Task=`03e9a479-1232-442f-9b24-4653460e0153`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:19:53,818::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:19:53,820::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:19:53,821::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:19:53,822::task::1172::TaskManager.Task::(prepare) Task=`03e9a479-1232-442f-9b24-4653460e0153`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:19:53,822::task::588::TaskManager.Task::(_updateState) Task=`03e9a479-1232-442f-9b24-4653460e0153`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:19:53,822::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:19:53,822::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:19:53,823::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:19:53,823::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:19:53,823::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:19:53,823::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:19:53,824::task::978::TaskManager.Task::(_decref) Task=`03e9a479-1232-442f-9b24-4653460e0153`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:19:53,824::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:53,824::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:53,825::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:19:53,825::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2226::DEBUG::2012-08-30 13:19:54,683::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2226::DEBUG::2012-08-30 13:19:54,683::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:19:55,915::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2227::DEBUG::2012-08-30 13:19:56,253::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2227::DEBUG::2012-08-30 13:19:56,254::task::588::TaskManager.Task::(_updateState) Task=`832c4dc4-1c66-41bf-bdd6-4872327d4dda`::moving from state init -> state preparing
>Thread-2227::INFO::2012-08-30 13:19:56,254::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2227::INFO::2012-08-30 13:19:56,254::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2227::DEBUG::2012-08-30 13:19:56,254::task::1172::TaskManager.Task::(prepare) Task=`832c4dc4-1c66-41bf-bdd6-4872327d4dda`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2227::DEBUG::2012-08-30 13:19:56,255::task::588::TaskManager.Task::(_updateState) Task=`832c4dc4-1c66-41bf-bdd6-4872327d4dda`::moving from state preparing -> state finished
>Thread-2227::DEBUG::2012-08-30 13:19:56,255::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2227::DEBUG::2012-08-30 13:19:56,255::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2227::DEBUG::2012-08-30 13:19:56,255::task::978::TaskManager.Task::(_decref) Task=`832c4dc4-1c66-41bf-bdd6-4872327d4dda`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:19:56,543::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:19:58,826::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2228::DEBUG::2012-08-30 13:19:59,423::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2228::DEBUG::2012-08-30 13:19:59,423::task::588::TaskManager.Task::(_updateState) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::moving from state init -> state preparing
>Thread-2228::INFO::2012-08-30 13:19:59,424::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2228::DEBUG::2012-08-30 13:19:59,424::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`22e1306f-997e-4b56-b68b-e3466cf4d337`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2228::DEBUG::2012-08-30 13:19:59,425::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2228::DEBUG::2012-08-30 13:19:59,425::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2228::DEBUG::2012-08-30 13:19:59,425::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`22e1306f-997e-4b56-b68b-e3466cf4d337`::Granted request
>Thread-2228::DEBUG::2012-08-30 13:19:59,426::task::817::TaskManager.Task::(resourceAcquired) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2228::DEBUG::2012-08-30 13:19:59,426::task::978::TaskManager.Task::(_decref) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::ref 1 aborting False
>Thread-2228::INFO::2012-08-30 13:19:59,429::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2228::DEBUG::2012-08-30 13:19:59,429::task::1172::TaskManager.Task::(prepare) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2228::DEBUG::2012-08-30 13:19:59,429::task::588::TaskManager.Task::(_updateState) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::moving from state preparing -> state finished
>Thread-2228::DEBUG::2012-08-30 13:19:59,429::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2228::DEBUG::2012-08-30 13:19:59,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2228::DEBUG::2012-08-30 13:19:59,430::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2228::DEBUG::2012-08-30 13:19:59,430::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2228::DEBUG::2012-08-30 13:19:59,430::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2228::DEBUG::2012-08-30 13:19:59,431::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2228::DEBUG::2012-08-30 13:19:59,431::task::978::TaskManager.Task::(_decref) Task=`65f36f58-81e0-458c-a951-26abfc244f29`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:20:00,916::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:00,917::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:20:03,550::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:20:03,827::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:05,918::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:08,828::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:08,829::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:10,919::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:13,830::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2232::DEBUG::2012-08-30 13:20:14,656::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2232::DEBUG::2012-08-30 13:20:14,657::task::588::TaskManager.Task::(_updateState) Task=`434d9542-c7e0-4488-860a-93cc272b376f`::moving from state init -> state preparing
>Thread-2232::INFO::2012-08-30 13:20:14,657::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2232::INFO::2012-08-30 13:20:14,657::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2232::DEBUG::2012-08-30 13:20:14,657::task::1172::TaskManager.Task::(prepare) Task=`434d9542-c7e0-4488-860a-93cc272b376f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2232::DEBUG::2012-08-30 13:20:14,658::task::588::TaskManager.Task::(_updateState) Task=`434d9542-c7e0-4488-860a-93cc272b376f`::moving from state preparing -> state finished
>Thread-2232::DEBUG::2012-08-30 13:20:14,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2232::DEBUG::2012-08-30 13:20:14,658::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2232::DEBUG::2012-08-30 13:20:14,658::task::978::TaskManager.Task::(_decref) Task=`434d9542-c7e0-4488-860a-93cc272b376f`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:20:15,920::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:15,920::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2234::DEBUG::2012-08-30 13:20:17,819::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2234::DEBUG::2012-08-30 13:20:17,819::task::588::TaskManager.Task::(_updateState) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::moving from state init -> state preparing
>Thread-2234::INFO::2012-08-30 13:20:17,820::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2234::DEBUG::2012-08-30 13:20:17,820::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ad7c4be3-32e1-43cc-86d6-90d0f8da26c0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2234::DEBUG::2012-08-30 13:20:17,820::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2234::DEBUG::2012-08-30 13:20:17,820::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2234::DEBUG::2012-08-30 13:20:17,821::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ad7c4be3-32e1-43cc-86d6-90d0f8da26c0`::Granted request
>Thread-2234::DEBUG::2012-08-30 13:20:17,821::task::817::TaskManager.Task::(resourceAcquired) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2234::DEBUG::2012-08-30 13:20:17,821::task::978::TaskManager.Task::(_decref) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::ref 1 aborting False
>Thread-2234::INFO::2012-08-30 13:20:17,824::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2234::DEBUG::2012-08-30 13:20:17,824::task::1172::TaskManager.Task::(prepare) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2234::DEBUG::2012-08-30 13:20:17,825::task::588::TaskManager.Task::(_updateState) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::moving from state preparing -> state finished
>Thread-2234::DEBUG::2012-08-30 13:20:17,825::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2234::DEBUG::2012-08-30 13:20:17,825::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2234::DEBUG::2012-08-30 13:20:17,825::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2234::DEBUG::2012-08-30 13:20:17,826::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2234::DEBUG::2012-08-30 13:20:17,826::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2234::DEBUG::2012-08-30 13:20:17,826::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2234::DEBUG::2012-08-30 13:20:17,826::task::978::TaskManager.Task::(_decref) Task=`d03e8c8b-0324-400d-ab29-afb75584d464`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:20:18,831::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2235::DEBUG::2012-08-30 13:20:19,405::task::588::TaskManager.Task::(_updateState) Task=`20ef9148-7bdf-478b-a1e9-c6990cb7d482`::moving from state init -> state preparing
>Thread-2235::INFO::2012-08-30 13:20:19,405::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2235::INFO::2012-08-30 13:20:19,406::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346304011.423502, 'code': 0, 'valid': True}}
>Thread-2235::DEBUG::2012-08-30 13:20:19,406::task::1172::TaskManager.Task::(prepare) Task=`20ef9148-7bdf-478b-a1e9-c6990cb7d482`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162601470947', 'lastCheck': 1346304011.423502, 'code': 0, 'valid': True}}
>Thread-2235::DEBUG::2012-08-30 13:20:19,406::task::588::TaskManager.Task::(_updateState) Task=`20ef9148-7bdf-478b-a1e9-c6990cb7d482`::moving from state preparing -> state finished
>Thread-2235::DEBUG::2012-08-30 13:20:19,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2235::DEBUG::2012-08-30 13:20:19,407::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2235::DEBUG::2012-08-30 13:20:19,407::task::978::TaskManager.Task::(_decref) Task=`20ef9148-7bdf-478b-a1e9-c6990cb7d482`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:20:20,921::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2236::DEBUG::2012-08-30 13:20:22,605::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2236::DEBUG::2012-08-30 13:20:22,605::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:20:23,833::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:23,833::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:25,922::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:20:26,571::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:20:28,835::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:20:30,923::task::588::TaskManager.Task::(_updateState) Task=`def63159-06e2-437d-831f-fe244bdca503`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:20:30,923::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:20:30,923::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`cab92603-a281-43a6-9e8a-27b922366e1c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:20:30,924::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:20:30,924::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:20:30,924::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`cab92603-a281-43a6-9e8a-27b922366e1c`::Granted request
>Thread-104::DEBUG::2012-08-30 13:20:30,925::task::817::TaskManager.Task::(resourceAcquired) Task=`def63159-06e2-437d-831f-fe244bdca503`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:20:30,925::task::978::TaskManager.Task::(_decref) Task=`def63159-06e2-437d-831f-fe244bdca503`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:20:30,926::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:20:30,928::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:20:30,930::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:20:30,930::task::1172::TaskManager.Task::(prepare) Task=`def63159-06e2-437d-831f-fe244bdca503`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:20:30,930::task::588::TaskManager.Task::(_updateState) Task=`def63159-06e2-437d-831f-fe244bdca503`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:20:30,930::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:20:30,930::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:20:30,931::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:20:30,931::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:20:30,931::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:20:30,931::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:20:30,932::task::978::TaskManager.Task::(_decref) Task=`def63159-06e2-437d-831f-fe244bdca503`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:20:30,932::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:30,932::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:30,933::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:30,933::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2238::DEBUG::2012-08-30 13:20:31,022::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2238::DEBUG::2012-08-30 13:20:31,023::task::588::TaskManager.Task::(_updateState) Task=`560f5901-3721-4c6d-b6b6-c681b87bf9a7`::moving from state init -> state preparing
>Thread-2238::INFO::2012-08-30 13:20:31,023::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2238::INFO::2012-08-30 13:20:31,023::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2238::DEBUG::2012-08-30 13:20:31,023::task::1172::TaskManager.Task::(prepare) Task=`560f5901-3721-4c6d-b6b6-c681b87bf9a7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2238::DEBUG::2012-08-30 13:20:31,024::task::588::TaskManager.Task::(_updateState) Task=`560f5901-3721-4c6d-b6b6-c681b87bf9a7`::moving from state preparing -> state finished
>Thread-2238::DEBUG::2012-08-30 13:20:31,024::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2238::DEBUG::2012-08-30 13:20:31,024::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2238::DEBUG::2012-08-30 13:20:31,024::task::978::TaskManager.Task::(_decref) Task=`560f5901-3721-4c6d-b6b6-c681b87bf9a7`::ref 0 aborting False
>Thread-26::DEBUG::2012-08-30 13:20:31,429::domainMonitor::164::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:20:31,430::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-26::DEBUG::2012-08-30 13:20:31,430::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 32fcf211-5c90-4362-911c-36bbc97a8a91' (cwd None)
>Thread-26::DEBUG::2012-08-30 13:20:31,539::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = '  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found\n'; <rc> = 5
>Thread-26::WARNING::2012-08-30 13:20:31,539::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "32fcf211-5c90-4362-911c-36bbc97a8a91" not found']
>Thread-26::DEBUG::2012-08-30 13:20:31,540::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-26::DEBUG::2012-08-30 13:20:31,546::fileSD::105::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.66.5.12:_mnt_data_rhevm/32fcf211-5c90-4362-911c-36bbc97a8a91
>Thread-26::DEBUG::2012-08-30 13:20:31,546::persistentDict::185::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend
>Thread-26::DEBUG::2012-08-30 13:20:31,549::persistentDict::226::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=512', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=whuang2', 'POOL_DOMAINS=32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=3', 'POOL_UUID=b25c9a7d-4634-406d-9881-bab3e1793946', 'REMOTE_PATH=10.66.5.12:/mnt/data/rhevm', 'ROLE=Master', 'SDUUID=32fcf211-5c90-4362-911c-36bbc97a8a91', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=33799b189a12a772f9c72f38aa351c173b84300d']
>Thread-26::DEBUG::2012-08-30 13:20:31,550::fileSD::389::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-26::WARNING::2012-08-30 13:20:31,550::sd::317::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_imageNS already registered
>Thread-26::WARNING::2012-08-30 13:20:31,550::sd::323::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 32fcf211-5c90-4362-911c-36bbc97a8a91_volumeNS already registered
>VM Channels Listener::DEBUG::2012-08-30 13:20:33,578::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:20:33,836::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2240::DEBUG::2012-08-30 13:20:34,327::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2240::DEBUG::2012-08-30 13:20:34,327::task::588::TaskManager.Task::(_updateState) Task=`9da5d875-1de8-4458-ab32-97d215720261`::moving from state init -> state preparing
>Thread-2240::INFO::2012-08-30 13:20:34,327::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2240::DEBUG::2012-08-30 13:20:34,328::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f4b03537-fbc8-4a71-9765-3b04b150fb0f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2240::DEBUG::2012-08-30 13:20:34,328::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2240::DEBUG::2012-08-30 13:20:34,328::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2240::DEBUG::2012-08-30 13:20:34,329::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f4b03537-fbc8-4a71-9765-3b04b150fb0f`::Granted request
>Thread-2240::DEBUG::2012-08-30 13:20:34,329::task::817::TaskManager.Task::(resourceAcquired) Task=`9da5d875-1de8-4458-ab32-97d215720261`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2240::DEBUG::2012-08-30 13:20:34,329::task::978::TaskManager.Task::(_decref) Task=`9da5d875-1de8-4458-ab32-97d215720261`::ref 1 aborting False
>Thread-2240::INFO::2012-08-30 13:20:34,332::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2240::DEBUG::2012-08-30 13:20:34,332::task::1172::TaskManager.Task::(prepare) Task=`9da5d875-1de8-4458-ab32-97d215720261`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2240::DEBUG::2012-08-30 13:20:34,333::task::588::TaskManager.Task::(_updateState) Task=`9da5d875-1de8-4458-ab32-97d215720261`::moving from state preparing -> state finished
>Thread-2240::DEBUG::2012-08-30 13:20:34,333::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2240::DEBUG::2012-08-30 13:20:34,333::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2240::DEBUG::2012-08-30 13:20:34,333::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2240::DEBUG::2012-08-30 13:20:34,334::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2240::DEBUG::2012-08-30 13:20:34,334::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2240::DEBUG::2012-08-30 13:20:34,334::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2240::DEBUG::2012-08-30 13:20:34,334::task::978::TaskManager.Task::(_decref) Task=`9da5d875-1de8-4458-ab32-97d215720261`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:20:35,934::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:38,837::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:38,838::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:40,935::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:43,839::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:45,936::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:45,937::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2248::DEBUG::2012-08-30 13:20:47,578::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2249::DEBUG::2012-08-30 13:20:47,578::task::588::TaskManager.Task::(_updateState) Task=`8b500c3f-bc1e-41cc-b2fb-826598deefd3`::moving from state init -> state preparing
>Thread-2248::DEBUG::2012-08-30 13:20:47,578::task::588::TaskManager.Task::(_updateState) Task=`e74172cf-89cf-478f-a9b0-8f7fb6d1be9b`::moving from state init -> state preparing
>Thread-2249::INFO::2012-08-30 13:20:47,579::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2248::INFO::2012-08-30 13:20:47,579::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2249::INFO::2012-08-30 13:20:47,579::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163316726685', 'lastCheck': 1346304041.563174, 'code': 0, 'valid': True}}
>Thread-2248::INFO::2012-08-30 13:20:47,579::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2249::DEBUG::2012-08-30 13:20:47,580::task::1172::TaskManager.Task::(prepare) Task=`8b500c3f-bc1e-41cc-b2fb-826598deefd3`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163316726685', 'lastCheck': 1346304041.563174, 'code': 0, 'valid': True}}
>Thread-2248::DEBUG::2012-08-30 13:20:47,580::task::1172::TaskManager.Task::(prepare) Task=`e74172cf-89cf-478f-a9b0-8f7fb6d1be9b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2249::DEBUG::2012-08-30 13:20:47,580::task::588::TaskManager.Task::(_updateState) Task=`8b500c3f-bc1e-41cc-b2fb-826598deefd3`::moving from state preparing -> state finished
>Thread-2248::DEBUG::2012-08-30 13:20:47,580::task::588::TaskManager.Task::(_updateState) Task=`e74172cf-89cf-478f-a9b0-8f7fb6d1be9b`::moving from state preparing -> state finished
>Thread-2249::DEBUG::2012-08-30 13:20:47,580::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2248::DEBUG::2012-08-30 13:20:47,581::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2249::DEBUG::2012-08-30 13:20:47,581::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2248::DEBUG::2012-08-30 13:20:47,581::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2249::DEBUG::2012-08-30 13:20:47,581::task::978::TaskManager.Task::(_decref) Task=`8b500c3f-bc1e-41cc-b2fb-826598deefd3`::ref 0 aborting False
>Thread-2248::DEBUG::2012-08-30 13:20:47,581::task::978::TaskManager.Task::(_decref) Task=`e74172cf-89cf-478f-a9b0-8f7fb6d1be9b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:20:48,841::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2251::DEBUG::2012-08-30 13:20:50,756::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2250::DEBUG::2012-08-30 13:20:50,757::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2251::DEBUG::2012-08-30 13:20:50,757::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-2250::DEBUG::2012-08-30 13:20:50,758::task::588::TaskManager.Task::(_updateState) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::moving from state init -> state preparing
>Thread-2250::INFO::2012-08-30 13:20:50,759::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2250::DEBUG::2012-08-30 13:20:50,759::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ef0d8a06-133a-4d5d-a51c-5eec68151cad`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2250::DEBUG::2012-08-30 13:20:50,760::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2250::DEBUG::2012-08-30 13:20:50,760::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2250::DEBUG::2012-08-30 13:20:50,760::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ef0d8a06-133a-4d5d-a51c-5eec68151cad`::Granted request
>Thread-2250::DEBUG::2012-08-30 13:20:50,761::task::817::TaskManager.Task::(resourceAcquired) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2250::DEBUG::2012-08-30 13:20:50,761::task::978::TaskManager.Task::(_decref) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::ref 1 aborting False
>Thread-2250::INFO::2012-08-30 13:20:50,764::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2250::DEBUG::2012-08-30 13:20:50,764::task::1172::TaskManager.Task::(prepare) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2250::DEBUG::2012-08-30 13:20:50,764::task::588::TaskManager.Task::(_updateState) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::moving from state preparing -> state finished
>Thread-2250::DEBUG::2012-08-30 13:20:50,764::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2250::DEBUG::2012-08-30 13:20:50,765::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2250::DEBUG::2012-08-30 13:20:50,765::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2250::DEBUG::2012-08-30 13:20:50,765::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2250::DEBUG::2012-08-30 13:20:50,765::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2250::DEBUG::2012-08-30 13:20:50,766::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2250::DEBUG::2012-08-30 13:20:50,766::task::978::TaskManager.Task::(_decref) Task=`a35629bf-0969-4bb9-bd0a-38b2798e1e61`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:20:50,937::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:20:53,842::task::588::TaskManager.Task::(_updateState) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:20:53,842::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:20:53,843::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`96a22286-cea5-4376-9d23-1e7440a98683`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:20:53,843::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:20:53,843::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:20:53,843::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`96a22286-cea5-4376-9d23-1e7440a98683`::Granted request
>Thread-40::DEBUG::2012-08-30 13:20:53,844::task::817::TaskManager.Task::(resourceAcquired) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:20:53,844::task::978::TaskManager.Task::(_decref) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:20:53,846::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:20:53,848::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:20:53,849::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:20:53,849::task::1172::TaskManager.Task::(prepare) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:20:53,849::task::588::TaskManager.Task::(_updateState) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:20:53,850::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:20:53,850::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:20:53,850::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:20:53,850::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:20:53,851::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:20:53,851::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:20:53,851::task::978::TaskManager.Task::(_decref) Task=`67d935c1-0fcd-4381-b333-67b1a90cbe1b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:20:53,851::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:53,852::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:53,852::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:20:53,853::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:20:55,938::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:20:56,600::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:20:58,854::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:00,939::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:00,940::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:21:03,608::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:21:03,855::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2254::DEBUG::2012-08-30 13:21:04,008::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2254::DEBUG::2012-08-30 13:21:04,009::task::588::TaskManager.Task::(_updateState) Task=`6e98d007-39da-4efa-83f2-ecbc46bf7070`::moving from state init -> state preparing
>Thread-2254::INFO::2012-08-30 13:21:04,009::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2254::INFO::2012-08-30 13:21:04,009::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2254::DEBUG::2012-08-30 13:21:04,010::task::1172::TaskManager.Task::(prepare) Task=`6e98d007-39da-4efa-83f2-ecbc46bf7070`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2254::DEBUG::2012-08-30 13:21:04,010::task::588::TaskManager.Task::(_updateState) Task=`6e98d007-39da-4efa-83f2-ecbc46bf7070`::moving from state preparing -> state finished
>Thread-2254::DEBUG::2012-08-30 13:21:04,010::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2254::DEBUG::2012-08-30 13:21:04,010::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2254::DEBUG::2012-08-30 13:21:04,010::task::978::TaskManager.Task::(_decref) Task=`6e98d007-39da-4efa-83f2-ecbc46bf7070`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:21:05,941::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2256::DEBUG::2012-08-30 13:21:07,341::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2256::DEBUG::2012-08-30 13:21:07,342::task::588::TaskManager.Task::(_updateState) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::moving from state init -> state preparing
>Thread-2256::INFO::2012-08-30 13:21:07,342::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2256::DEBUG::2012-08-30 13:21:07,342::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c9ebba51-5be7-4010-ab19-f94d73d59c4c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2256::DEBUG::2012-08-30 13:21:07,343::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2256::DEBUG::2012-08-30 13:21:07,343::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2256::DEBUG::2012-08-30 13:21:07,343::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`c9ebba51-5be7-4010-ab19-f94d73d59c4c`::Granted request
>Thread-2256::DEBUG::2012-08-30 13:21:07,343::task::817::TaskManager.Task::(resourceAcquired) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2256::DEBUG::2012-08-30 13:21:07,344::task::978::TaskManager.Task::(_decref) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::ref 1 aborting False
>Thread-2256::INFO::2012-08-30 13:21:07,347::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2256::DEBUG::2012-08-30 13:21:07,347::task::1172::TaskManager.Task::(prepare) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2256::DEBUG::2012-08-30 13:21:07,347::task::588::TaskManager.Task::(_updateState) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::moving from state preparing -> state finished
>Thread-2256::DEBUG::2012-08-30 13:21:07,347::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2256::DEBUG::2012-08-30 13:21:07,348::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2256::DEBUG::2012-08-30 13:21:07,349::task::978::TaskManager.Task::(_decref) Task=`bd6b476b-52ce-4dd3-a73c-1c71bb1cdb6b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:21:08,856::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:21:08,857::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:10,942::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:21:13,858::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:15,943::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:15,943::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2258::DEBUG::2012-08-30 13:21:16,744::task::588::TaskManager.Task::(_updateState) Task=`bddd6ce6-845d-4571-9b90-1b20f174e9d5`::moving from state init -> state preparing
>Thread-2258::INFO::2012-08-30 13:21:16,744::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2258::INFO::2012-08-30 13:21:16,745::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163388252258', 'lastCheck': 1346304071.583734, 'code': 0, 'valid': True}}
>Thread-2258::DEBUG::2012-08-30 13:21:16,745::task::1172::TaskManager.Task::(prepare) Task=`bddd6ce6-845d-4571-9b90-1b20f174e9d5`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163388252258', 'lastCheck': 1346304071.583734, 'code': 0, 'valid': True}}
>Thread-2258::DEBUG::2012-08-30 13:21:16,745::task::588::TaskManager.Task::(_updateState) Task=`bddd6ce6-845d-4571-9b90-1b20f174e9d5`::moving from state preparing -> state finished
>Thread-2258::DEBUG::2012-08-30 13:21:16,745::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2258::DEBUG::2012-08-30 13:21:16,745::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2258::DEBUG::2012-08-30 13:21:16,746::task::978::TaskManager.Task::(_decref) Task=`bddd6ce6-845d-4571-9b90-1b20f174e9d5`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:21:18,859::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2259::DEBUG::2012-08-30 13:21:19,935::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2260::DEBUG::2012-08-30 13:21:19,936::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2259::DEBUG::2012-08-30 13:21:19,936::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-2260::DEBUG::2012-08-30 13:21:19,937::task::588::TaskManager.Task::(_updateState) Task=`1b37c8a5-78f6-4279-ad3d-b8171072e663`::moving from state init -> state preparing
>Thread-2260::INFO::2012-08-30 13:21:19,938::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2260::INFO::2012-08-30 13:21:19,938::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2260::DEBUG::2012-08-30 13:21:19,939::task::1172::TaskManager.Task::(prepare) Task=`1b37c8a5-78f6-4279-ad3d-b8171072e663`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2260::DEBUG::2012-08-30 13:21:19,939::task::588::TaskManager.Task::(_updateState) Task=`1b37c8a5-78f6-4279-ad3d-b8171072e663`::moving from state preparing -> state finished
>Thread-2260::DEBUG::2012-08-30 13:21:19,939::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2260::DEBUG::2012-08-30 13:21:19,939::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2260::DEBUG::2012-08-30 13:21:19,939::task::978::TaskManager.Task::(_decref) Task=`1b37c8a5-78f6-4279-ad3d-b8171072e663`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:21:20,944::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2261::DEBUG::2012-08-30 13:21:23,121::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2261::DEBUG::2012-08-30 13:21:23,121::task::588::TaskManager.Task::(_updateState) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::moving from state init -> state preparing
>Thread-2261::INFO::2012-08-30 13:21:23,122::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2261::DEBUG::2012-08-30 13:21:23,122::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f06a4aad-502e-49ba-bc76-a592c4c9d9bc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2261::DEBUG::2012-08-30 13:21:23,122::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2261::DEBUG::2012-08-30 13:21:23,122::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2261::DEBUG::2012-08-30 13:21:23,123::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`f06a4aad-502e-49ba-bc76-a592c4c9d9bc`::Granted request
>Thread-2261::DEBUG::2012-08-30 13:21:23,123::task::817::TaskManager.Task::(resourceAcquired) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2261::DEBUG::2012-08-30 13:21:23,123::task::978::TaskManager.Task::(_decref) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::ref 1 aborting False
>Thread-2261::INFO::2012-08-30 13:21:23,126::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2261::DEBUG::2012-08-30 13:21:23,126::task::1172::TaskManager.Task::(prepare) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2261::DEBUG::2012-08-30 13:21:23,127::task::588::TaskManager.Task::(_updateState) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::moving from state preparing -> state finished
>Thread-2261::DEBUG::2012-08-30 13:21:23,127::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2261::DEBUG::2012-08-30 13:21:23,127::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2261::DEBUG::2012-08-30 13:21:23,127::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2261::DEBUG::2012-08-30 13:21:23,128::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2261::DEBUG::2012-08-30 13:21:23,128::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2261::DEBUG::2012-08-30 13:21:23,128::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2261::DEBUG::2012-08-30 13:21:23,128::task::978::TaskManager.Task::(_decref) Task=`0d09cb0a-9753-44fc-8fa1-e2b37dfbefe6`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:21:23,861::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:21:23,861::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:25,945::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:21:26,627::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:21:28,862::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:21:30,946::task::588::TaskManager.Task::(_updateState) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:21:30,946::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:21:30,947::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a5826bcf-9087-49c5-b973-1e34eb3632a5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:21:30,947::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:21:30,947::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:21:30,947::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`a5826bcf-9087-49c5-b973-1e34eb3632a5`::Granted request
>Thread-104::DEBUG::2012-08-30 13:21:30,948::task::817::TaskManager.Task::(resourceAcquired) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:21:30,948::task::978::TaskManager.Task::(_decref) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:21:30,949::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:21:30,952::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:21:30,953::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:21:30,953::task::1172::TaskManager.Task::(prepare) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:21:30,953::task::588::TaskManager.Task::(_updateState) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:21:30,953::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:21:30,954::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:21:30,954::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:21:30,954::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:21:30,954::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:21:30,955::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:21:30,955::task::978::TaskManager.Task::(_decref) Task=`e72bdb30-eab6-4528-8391-3e51ec90c918`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:21:30,955::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:30,956::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:30,956::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:30,957::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:21:33,632::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:21:33,864::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:35,957::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2265::DEBUG::2012-08-30 13:21:36,328::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2265::DEBUG::2012-08-30 13:21:36,328::task::588::TaskManager.Task::(_updateState) Task=`3e775112-f3d1-4211-baf8-66bd1f92b12b`::moving from state init -> state preparing
>Thread-2265::INFO::2012-08-30 13:21:36,329::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2265::INFO::2012-08-30 13:21:36,329::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2265::DEBUG::2012-08-30 13:21:36,329::task::1172::TaskManager.Task::(prepare) Task=`3e775112-f3d1-4211-baf8-66bd1f92b12b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2265::DEBUG::2012-08-30 13:21:36,329::task::588::TaskManager.Task::(_updateState) Task=`3e775112-f3d1-4211-baf8-66bd1f92b12b`::moving from state preparing -> state finished
>Thread-2265::DEBUG::2012-08-30 13:21:36,330::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2265::DEBUG::2012-08-30 13:21:36,330::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2265::DEBUG::2012-08-30 13:21:36,330::task::978::TaskManager.Task::(_decref) Task=`3e775112-f3d1-4211-baf8-66bd1f92b12b`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:21:38,865::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:21:38,865::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2266::DEBUG::2012-08-30 13:21:39,197::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2266::DEBUG::2012-08-30 13:21:39,197::task::588::TaskManager.Task::(_updateState) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::moving from state init -> state preparing
>Thread-2266::INFO::2012-08-30 13:21:39,197::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2266::DEBUG::2012-08-30 13:21:39,198::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6047fbad-ffe1-4f76-baae-3414da2366c6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2266::DEBUG::2012-08-30 13:21:39,198::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2266::DEBUG::2012-08-30 13:21:39,198::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2266::DEBUG::2012-08-30 13:21:39,198::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`6047fbad-ffe1-4f76-baae-3414da2366c6`::Granted request
>Thread-2266::DEBUG::2012-08-30 13:21:39,199::task::817::TaskManager.Task::(resourceAcquired) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2266::DEBUG::2012-08-30 13:21:39,199::task::978::TaskManager.Task::(_decref) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::ref 1 aborting False
>Thread-2266::INFO::2012-08-30 13:21:39,202::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2266::DEBUG::2012-08-30 13:21:39,202::task::1172::TaskManager.Task::(prepare) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2266::DEBUG::2012-08-30 13:21:39,203::task::588::TaskManager.Task::(_updateState) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::moving from state preparing -> state finished
>Thread-2266::DEBUG::2012-08-30 13:21:39,203::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2266::DEBUG::2012-08-30 13:21:39,203::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2266::DEBUG::2012-08-30 13:21:39,203::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2266::DEBUG::2012-08-30 13:21:39,203::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2266::DEBUG::2012-08-30 13:21:39,204::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2266::DEBUG::2012-08-30 13:21:39,204::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2266::DEBUG::2012-08-30 13:21:39,204::task::978::TaskManager.Task::(_decref) Task=`d48b24b4-b8cc-4269-a7e9-9e170a241181`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:21:40,958::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:21:43,866::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:45,959::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:45,960::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2268::DEBUG::2012-08-30 13:21:46,182::task::588::TaskManager.Task::(_updateState) Task=`bfe4028f-757a-4a45-99f7-3846eb923b09`::moving from state init -> state preparing
>Thread-2268::INFO::2012-08-30 13:21:46,182::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2268::INFO::2012-08-30 13:21:46,182::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162410736084', 'lastCheck': 1346304101.602519, 'code': 0, 'valid': True}}
>Thread-2268::DEBUG::2012-08-30 13:21:46,182::task::1172::TaskManager.Task::(prepare) Task=`bfe4028f-757a-4a45-99f7-3846eb923b09`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162410736084', 'lastCheck': 1346304101.602519, 'code': 0, 'valid': True}}
>Thread-2268::DEBUG::2012-08-30 13:21:46,183::task::588::TaskManager.Task::(_updateState) Task=`bfe4028f-757a-4a45-99f7-3846eb923b09`::moving from state preparing -> state finished
>Thread-2268::DEBUG::2012-08-30 13:21:46,183::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2268::DEBUG::2012-08-30 13:21:46,183::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2268::DEBUG::2012-08-30 13:21:46,183::task::978::TaskManager.Task::(_decref) Task=`bfe4028f-757a-4a45-99f7-3846eb923b09`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:21:48,868::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2269::DEBUG::2012-08-30 13:21:49,660::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2269::DEBUG::2012-08-30 13:21:49,660::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:21:50,961::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2270::DEBUG::2012-08-30 13:21:53,640::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2270::DEBUG::2012-08-30 13:21:53,640::task::588::TaskManager.Task::(_updateState) Task=`0ec0d9ec-3650-4ec4-ad79-8abf10f4bf9a`::moving from state init -> state preparing
>Thread-2270::INFO::2012-08-30 13:21:53,640::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2270::INFO::2012-08-30 13:21:53,641::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2270::DEBUG::2012-08-30 13:21:53,641::task::1172::TaskManager.Task::(prepare) Task=`0ec0d9ec-3650-4ec4-ad79-8abf10f4bf9a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2270::DEBUG::2012-08-30 13:21:53,641::task::588::TaskManager.Task::(_updateState) Task=`0ec0d9ec-3650-4ec4-ad79-8abf10f4bf9a`::moving from state preparing -> state finished
>Thread-2270::DEBUG::2012-08-30 13:21:53,641::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2270::DEBUG::2012-08-30 13:21:53,642::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2270::DEBUG::2012-08-30 13:21:53,642::task::978::TaskManager.Task::(_decref) Task=`0ec0d9ec-3650-4ec4-ad79-8abf10f4bf9a`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 13:21:53,869::task::588::TaskManager.Task::(_updateState) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:21:53,869::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:21:53,870::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`37daa529-999e-4cea-a2e9-1b57673419dc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:21:53,870::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:21:53,870::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:21:53,870::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`37daa529-999e-4cea-a2e9-1b57673419dc`::Granted request
>Thread-40::DEBUG::2012-08-30 13:21:53,871::task::817::TaskManager.Task::(resourceAcquired) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:21:53,871::task::978::TaskManager.Task::(_decref) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:21:53,873::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:21:53,875::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:21:53,876::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:21:53,876::task::1172::TaskManager.Task::(prepare) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:21:53,876::task::588::TaskManager.Task::(_updateState) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:21:53,877::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:21:53,878::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:21:53,878::task::978::TaskManager.Task::(_decref) Task=`8e4f7490-f9f4-4b41-8a90-76302b3d7df9`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:21:53,878::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:21:53,879::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:21:53,879::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:21:53,880::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:21:55,962::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:21:56,654::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2272::DEBUG::2012-08-30 13:21:57,359::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2272::DEBUG::2012-08-30 13:21:57,360::task::588::TaskManager.Task::(_updateState) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::moving from state init -> state preparing
>Thread-2272::INFO::2012-08-30 13:21:57,360::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2272::DEBUG::2012-08-30 13:21:57,360::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d5867a92-a3f1-4b04-aad2-45d46223ede0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2272::DEBUG::2012-08-30 13:21:57,360::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2272::DEBUG::2012-08-30 13:21:57,361::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2272::DEBUG::2012-08-30 13:21:57,361::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`d5867a92-a3f1-4b04-aad2-45d46223ede0`::Granted request
>Thread-2272::DEBUG::2012-08-30 13:21:57,361::task::817::TaskManager.Task::(resourceAcquired) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2272::DEBUG::2012-08-30 13:21:57,362::task::978::TaskManager.Task::(_decref) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::ref 1 aborting False
>Thread-2272::INFO::2012-08-30 13:21:57,364::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2272::DEBUG::2012-08-30 13:21:57,365::task::1172::TaskManager.Task::(prepare) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2272::DEBUG::2012-08-30 13:21:57,365::task::588::TaskManager.Task::(_updateState) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::moving from state preparing -> state finished
>Thread-2272::DEBUG::2012-08-30 13:21:57,365::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2272::DEBUG::2012-08-30 13:21:57,365::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2272::DEBUG::2012-08-30 13:21:57,366::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2272::DEBUG::2012-08-30 13:21:57,366::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2272::DEBUG::2012-08-30 13:21:57,366::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2272::DEBUG::2012-08-30 13:21:57,366::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2272::DEBUG::2012-08-30 13:21:57,367::task::978::TaskManager.Task::(_decref) Task=`123a447a-b801-4f2d-aec1-bac76fbf0816`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:21:58,881::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:00,963::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:00,963::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:22:03,660::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:22:03,882::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:05,964::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:08,883::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:08,884::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2275::DEBUG::2012-08-30 13:22:10,831::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2275::DEBUG::2012-08-30 13:22:10,832::task::588::TaskManager.Task::(_updateState) Task=`d18f7942-f50e-4910-bf77-1de0c7eb92a1`::moving from state init -> state preparing
>Thread-2275::INFO::2012-08-30 13:22:10,832::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2275::INFO::2012-08-30 13:22:10,832::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2275::DEBUG::2012-08-30 13:22:10,833::task::1172::TaskManager.Task::(prepare) Task=`d18f7942-f50e-4910-bf77-1de0c7eb92a1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2275::DEBUG::2012-08-30 13:22:10,833::task::588::TaskManager.Task::(_updateState) Task=`d18f7942-f50e-4910-bf77-1de0c7eb92a1`::moving from state preparing -> state finished
>Thread-2275::DEBUG::2012-08-30 13:22:10,833::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2275::DEBUG::2012-08-30 13:22:10,833::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2275::DEBUG::2012-08-30 13:22:10,834::task::978::TaskManager.Task::(_decref) Task=`d18f7942-f50e-4910-bf77-1de0c7eb92a1`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:22:10,965::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:13,885::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2277::DEBUG::2012-08-30 13:22:14,213::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2277::DEBUG::2012-08-30 13:22:14,214::task::588::TaskManager.Task::(_updateState) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::moving from state init -> state preparing
>Thread-2277::INFO::2012-08-30 13:22:14,214::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2277::DEBUG::2012-08-30 13:22:14,214::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9ac115c3-5e59-4535-a6bb-aabc1151e020`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2277::DEBUG::2012-08-30 13:22:14,214::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2277::DEBUG::2012-08-30 13:22:14,215::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2277::DEBUG::2012-08-30 13:22:14,215::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`9ac115c3-5e59-4535-a6bb-aabc1151e020`::Granted request
>Thread-2277::DEBUG::2012-08-30 13:22:14,215::task::817::TaskManager.Task::(resourceAcquired) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2277::DEBUG::2012-08-30 13:22:14,216::task::978::TaskManager.Task::(_decref) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::ref 1 aborting False
>Thread-2277::INFO::2012-08-30 13:22:14,218::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2277::DEBUG::2012-08-30 13:22:14,219::task::1172::TaskManager.Task::(prepare) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2277::DEBUG::2012-08-30 13:22:14,219::task::588::TaskManager.Task::(_updateState) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::moving from state preparing -> state finished
>Thread-2277::DEBUG::2012-08-30 13:22:14,219::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2277::DEBUG::2012-08-30 13:22:14,219::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2277::DEBUG::2012-08-30 13:22:14,220::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2277::DEBUG::2012-08-30 13:22:14,220::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2277::DEBUG::2012-08-30 13:22:14,220::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2277::DEBUG::2012-08-30 13:22:14,220::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2277::DEBUG::2012-08-30 13:22:14,221::task::978::TaskManager.Task::(_decref) Task=`0a099743-3ad9-44c6-b24e-9988a0c018b7`::ref 0 aborting False
>Thread-2278::DEBUG::2012-08-30 13:22:15,843::task::588::TaskManager.Task::(_updateState) Task=`0df545c5-2d73-48b1-b044-a17d0739e2cf`::moving from state init -> state preparing
>Thread-2278::INFO::2012-08-30 13:22:15,843::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2278::INFO::2012-08-30 13:22:15,843::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162982940674', 'lastCheck': 1346304131.6231959, 'code': 0, 'valid': True}}
>Thread-2278::DEBUG::2012-08-30 13:22:15,843::task::1172::TaskManager.Task::(prepare) Task=`0df545c5-2d73-48b1-b044-a17d0739e2cf`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162982940674', 'lastCheck': 1346304131.6231959, 'code': 0, 'valid': True}}
>Thread-2278::DEBUG::2012-08-30 13:22:15,844::task::588::TaskManager.Task::(_updateState) Task=`0df545c5-2d73-48b1-b044-a17d0739e2cf`::moving from state preparing -> state finished
>Thread-2278::DEBUG::2012-08-30 13:22:15,844::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2278::DEBUG::2012-08-30 13:22:15,844::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2278::DEBUG::2012-08-30 13:22:15,844::task::978::TaskManager.Task::(_decref) Task=`0df545c5-2d73-48b1-b044-a17d0739e2cf`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:22:15,966::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:15,966::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:18,886::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2279::DEBUG::2012-08-30 13:22:19,052::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2279::DEBUG::2012-08-30 13:22:19,053::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:22:20,967::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:23,887::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:23,888::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:25,968::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:22:26,682::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-2281::DEBUG::2012-08-30 13:22:27,479::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2281::DEBUG::2012-08-30 13:22:27,479::task::588::TaskManager.Task::(_updateState) Task=`a6c24215-778a-4f56-971d-5fc9c00b3831`::moving from state init -> state preparing
>Thread-2281::INFO::2012-08-30 13:22:27,480::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2281::INFO::2012-08-30 13:22:27,480::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2281::DEBUG::2012-08-30 13:22:27,480::task::1172::TaskManager.Task::(prepare) Task=`a6c24215-778a-4f56-971d-5fc9c00b3831`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2281::DEBUG::2012-08-30 13:22:27,480::task::588::TaskManager.Task::(_updateState) Task=`a6c24215-778a-4f56-971d-5fc9c00b3831`::moving from state preparing -> state finished
>Thread-2281::DEBUG::2012-08-30 13:22:27,481::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2281::DEBUG::2012-08-30 13:22:27,481::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2281::DEBUG::2012-08-30 13:22:27,481::task::978::TaskManager.Task::(_decref) Task=`a6c24215-778a-4f56-971d-5fc9c00b3831`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:22:28,889::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2283::DEBUG::2012-08-30 13:22:30,618::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2283::DEBUG::2012-08-30 13:22:30,619::task::588::TaskManager.Task::(_updateState) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::moving from state init -> state preparing
>Thread-2283::INFO::2012-08-30 13:22:30,619::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2283::DEBUG::2012-08-30 13:22:30,620::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`26a150fd-2e26-4e67-8ec3-7b277017f930`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2283::DEBUG::2012-08-30 13:22:30,620::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2283::DEBUG::2012-08-30 13:22:30,620::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2283::DEBUG::2012-08-30 13:22:30,620::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`26a150fd-2e26-4e67-8ec3-7b277017f930`::Granted request
>Thread-2283::DEBUG::2012-08-30 13:22:30,621::task::817::TaskManager.Task::(resourceAcquired) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2283::DEBUG::2012-08-30 13:22:30,621::task::978::TaskManager.Task::(_decref) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::ref 1 aborting False
>Thread-2283::INFO::2012-08-30 13:22:30,624::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2283::DEBUG::2012-08-30 13:22:30,624::task::1172::TaskManager.Task::(prepare) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2283::DEBUG::2012-08-30 13:22:30,624::task::588::TaskManager.Task::(_updateState) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::moving from state preparing -> state finished
>Thread-2283::DEBUG::2012-08-30 13:22:30,625::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2283::DEBUG::2012-08-30 13:22:30,625::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2283::DEBUG::2012-08-30 13:22:30,625::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2283::DEBUG::2012-08-30 13:22:30,625::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2283::DEBUG::2012-08-30 13:22:30,626::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2283::DEBUG::2012-08-30 13:22:30,626::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2283::DEBUG::2012-08-30 13:22:30,626::task::978::TaskManager.Task::(_decref) Task=`0fabd158-d89d-42f0-a262-7bb1ca3bbf79`::ref 0 aborting False
>Thread-104::DEBUG::2012-08-30 13:22:30,969::task::588::TaskManager.Task::(_updateState) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:22:30,969::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:22:30,970::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`09cc1736-a696-4ef4-b285-6b50d185a4be`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:22:30,970::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:22:30,970::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:22:30,970::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`09cc1736-a696-4ef4-b285-6b50d185a4be`::Granted request
>Thread-104::DEBUG::2012-08-30 13:22:30,971::task::817::TaskManager.Task::(resourceAcquired) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:22:30,971::task::978::TaskManager.Task::(_decref) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:22:30,972::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:22:30,975::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:22:30,976::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:22:30,976::task::1172::TaskManager.Task::(prepare) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:22:30,976::task::588::TaskManager.Task::(_updateState) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:22:30,976::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:22:30,977::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:22:30,978::task::978::TaskManager.Task::(_decref) Task=`f75f1f8f-e045-47dd-a1cb-9b684c836a99`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:22:30,978::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:30,979::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:30,979::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:30,979::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:22:33,688::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:22:33,890::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:35,980::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:38,892::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:38,892::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:40,981::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2286::DEBUG::2012-08-30 13:22:43,307::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2286::DEBUG::2012-08-30 13:22:43,308::task::588::TaskManager.Task::(_updateState) Task=`0da26833-98dc-422c-8a87-13cca0acaa5e`::moving from state init -> state preparing
>Thread-2286::INFO::2012-08-30 13:22:43,308::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2286::INFO::2012-08-30 13:22:43,308::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2286::DEBUG::2012-08-30 13:22:43,308::task::1172::TaskManager.Task::(prepare) Task=`0da26833-98dc-422c-8a87-13cca0acaa5e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2286::DEBUG::2012-08-30 13:22:43,309::task::588::TaskManager.Task::(_updateState) Task=`0da26833-98dc-422c-8a87-13cca0acaa5e`::moving from state preparing -> state finished
>Thread-2286::DEBUG::2012-08-30 13:22:43,309::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2286::DEBUG::2012-08-30 13:22:43,309::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2286::DEBUG::2012-08-30 13:22:43,309::task::978::TaskManager.Task::(_decref) Task=`0da26833-98dc-422c-8a87-13cca0acaa5e`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:22:43,893::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2287::DEBUG::2012-08-30 13:22:45,148::task::588::TaskManager.Task::(_updateState) Task=`a4755cba-2528-42f2-a828-4eb672dbcb01`::moving from state init -> state preparing
>Thread-2287::INFO::2012-08-30 13:22:45,148::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2287::INFO::2012-08-30 13:22:45,149::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346304161.6421461, 'code': 0, 'valid': True}}
>Thread-2287::DEBUG::2012-08-30 13:22:45,149::task::1172::TaskManager.Task::(prepare) Task=`a4755cba-2528-42f2-a828-4eb672dbcb01`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00162291526794', 'lastCheck': 1346304161.6421461, 'code': 0, 'valid': True}}
>Thread-2287::DEBUG::2012-08-30 13:22:45,149::task::588::TaskManager.Task::(_updateState) Task=`a4755cba-2528-42f2-a828-4eb672dbcb01`::moving from state preparing -> state finished
>Thread-2287::DEBUG::2012-08-30 13:22:45,149::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2287::DEBUG::2012-08-30 13:22:45,150::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2287::DEBUG::2012-08-30 13:22:45,150::task::978::TaskManager.Task::(_decref) Task=`a4755cba-2528-42f2-a828-4eb672dbcb01`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:22:45,982::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:45,983::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2288::DEBUG::2012-08-30 13:22:46,722::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2288::DEBUG::2012-08-30 13:22:46,723::task::588::TaskManager.Task::(_updateState) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::moving from state init -> state preparing
>Thread-2288::INFO::2012-08-30 13:22:46,723::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2288::DEBUG::2012-08-30 13:22:46,724::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`92da28b6-5e2d-4c0f-9138-e4a007d8ea8f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2288::DEBUG::2012-08-30 13:22:46,724::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2288::DEBUG::2012-08-30 13:22:46,724::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2288::DEBUG::2012-08-30 13:22:46,724::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`92da28b6-5e2d-4c0f-9138-e4a007d8ea8f`::Granted request
>Thread-2288::DEBUG::2012-08-30 13:22:46,725::task::817::TaskManager.Task::(resourceAcquired) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2288::DEBUG::2012-08-30 13:22:46,725::task::978::TaskManager.Task::(_decref) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::ref 1 aborting False
>Thread-2288::INFO::2012-08-30 13:22:46,728::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2288::DEBUG::2012-08-30 13:22:46,728::task::1172::TaskManager.Task::(prepare) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2288::DEBUG::2012-08-30 13:22:46,728::task::588::TaskManager.Task::(_updateState) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::moving from state preparing -> state finished
>Thread-2288::DEBUG::2012-08-30 13:22:46,728::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2288::DEBUG::2012-08-30 13:22:46,729::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2288::DEBUG::2012-08-30 13:22:46,729::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2288::DEBUG::2012-08-30 13:22:46,729::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2288::DEBUG::2012-08-30 13:22:46,729::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2288::DEBUG::2012-08-30 13:22:46,730::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2288::DEBUG::2012-08-30 13:22:46,730::task::978::TaskManager.Task::(_decref) Task=`bf81b85c-8fcc-4d4f-9794-573a83957cba`::ref 0 aborting False
>Thread-2289::DEBUG::2012-08-30 13:22:48,301::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2289::DEBUG::2012-08-30 13:22:48,302::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:22:48,895::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:50,984::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::DEBUG::2012-08-30 13:22:53,896::task::588::TaskManager.Task::(_updateState) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:22:53,896::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:22:53,897::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4fc1ced7-b915-492d-bca4-25d7c9367a46`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:22:53,897::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:22:53,897::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:22:53,897::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`4fc1ced7-b915-492d-bca4-25d7c9367a46`::Granted request
>Thread-40::DEBUG::2012-08-30 13:22:53,898::task::817::TaskManager.Task::(resourceAcquired) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:22:53,898::task::978::TaskManager.Task::(_decref) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:22:53,900::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:22:53,902::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:22:53,903::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:22:53,903::task::1172::TaskManager.Task::(prepare) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:22:53,903::task::588::TaskManager.Task::(_updateState) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:22:53,903::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:22:53,904::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:22:53,904::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:22:53,904::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:22:53,904::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:22:53,905::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:22:53,905::task::978::TaskManager.Task::(_decref) Task=`3151225a-f9a7-4ddf-8584-e650efe4389c`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:22:53,905::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:53,906::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:53,906::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:22:53,907::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:22:55,985::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:22:56,710::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:22:58,908::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2292::DEBUG::2012-08-30 13:22:59,949::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2292::DEBUG::2012-08-30 13:22:59,950::task::588::TaskManager.Task::(_updateState) Task=`11e1eb14-da86-4518-8410-66b0d21273c7`::moving from state init -> state preparing
>Thread-2292::INFO::2012-08-30 13:22:59,950::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2292::INFO::2012-08-30 13:22:59,950::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2292::DEBUG::2012-08-30 13:22:59,950::task::1172::TaskManager.Task::(prepare) Task=`11e1eb14-da86-4518-8410-66b0d21273c7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2292::DEBUG::2012-08-30 13:22:59,951::task::588::TaskManager.Task::(_updateState) Task=`11e1eb14-da86-4518-8410-66b0d21273c7`::moving from state preparing -> state finished
>Thread-2292::DEBUG::2012-08-30 13:22:59,951::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2292::DEBUG::2012-08-30 13:22:59,951::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2292::DEBUG::2012-08-30 13:22:59,951::task::978::TaskManager.Task::(_decref) Task=`11e1eb14-da86-4518-8410-66b0d21273c7`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:23:00,986::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:00,986::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2293::DEBUG::2012-08-30 13:23:03,146::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2293::DEBUG::2012-08-30 13:23:03,147::task::588::TaskManager.Task::(_updateState) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::moving from state init -> state preparing
>Thread-2293::INFO::2012-08-30 13:23:03,147::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2293::DEBUG::2012-08-30 13:23:03,147::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5950726f-36e5-4a47-8c47-5a3f20a0f673`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2293::DEBUG::2012-08-30 13:23:03,148::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2293::DEBUG::2012-08-30 13:23:03,148::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2293::DEBUG::2012-08-30 13:23:03,148::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`5950726f-36e5-4a47-8c47-5a3f20a0f673`::Granted request
>Thread-2293::DEBUG::2012-08-30 13:23:03,148::task::817::TaskManager.Task::(resourceAcquired) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2293::DEBUG::2012-08-30 13:23:03,149::task::978::TaskManager.Task::(_decref) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::ref 1 aborting False
>Thread-2293::INFO::2012-08-30 13:23:03,152::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2293::DEBUG::2012-08-30 13:23:03,152::task::1172::TaskManager.Task::(prepare) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2293::DEBUG::2012-08-30 13:23:03,152::task::588::TaskManager.Task::(_updateState) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::moving from state preparing -> state finished
>Thread-2293::DEBUG::2012-08-30 13:23:03,152::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2293::DEBUG::2012-08-30 13:23:03,153::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2293::DEBUG::2012-08-30 13:23:03,153::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2293::DEBUG::2012-08-30 13:23:03,153::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2293::DEBUG::2012-08-30 13:23:03,153::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2293::DEBUG::2012-08-30 13:23:03,154::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2293::DEBUG::2012-08-30 13:23:03,154::task::978::TaskManager.Task::(_decref) Task=`c7e731c7-ad47-4aef-92d5-6aed8a51f5d1`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:23:03,717::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:23:03,909::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:05,987::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:08,910::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:08,911::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:10,988::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2296::DEBUG::2012-08-30 13:23:13,456::task::588::TaskManager.Task::(_updateState) Task=`c3d01d39-8484-480f-b778-0f8b289c0e76`::moving from state init -> state preparing
>Thread-2296::INFO::2012-08-30 13:23:13,457::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2296::INFO::2012-08-30 13:23:13,457::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346304191.6629989, 'code': 0, 'valid': True}}
>Thread-2296::DEBUG::2012-08-30 13:23:13,457::task::1172::TaskManager.Task::(prepare) Task=`c3d01d39-8484-480f-b778-0f8b289c0e76`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00163102149963', 'lastCheck': 1346304191.6629989, 'code': 0, 'valid': True}}
>Thread-2296::DEBUG::2012-08-30 13:23:13,457::task::588::TaskManager.Task::(_updateState) Task=`c3d01d39-8484-480f-b778-0f8b289c0e76`::moving from state preparing -> state finished
>Thread-2296::DEBUG::2012-08-30 13:23:13,458::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2296::DEBUG::2012-08-30 13:23:13,458::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2296::DEBUG::2012-08-30 13:23:13,458::task::978::TaskManager.Task::(_decref) Task=`c3d01d39-8484-480f-b778-0f8b289c0e76`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:23:13,912::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:15,989::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:15,989::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2298::DEBUG::2012-08-30 13:23:16,388::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2297::DEBUG::2012-08-30 13:23:16,388::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2298::DEBUG::2012-08-30 13:23:16,389::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-2297::DEBUG::2012-08-30 13:23:16,389::task::588::TaskManager.Task::(_updateState) Task=`e3c943a9-f675-46c7-8811-db294e5cab73`::moving from state init -> state preparing
>Thread-2297::INFO::2012-08-30 13:23:16,390::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2297::INFO::2012-08-30 13:23:16,391::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2297::DEBUG::2012-08-30 13:23:16,391::task::1172::TaskManager.Task::(prepare) Task=`e3c943a9-f675-46c7-8811-db294e5cab73`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2297::DEBUG::2012-08-30 13:23:16,391::task::588::TaskManager.Task::(_updateState) Task=`e3c943a9-f675-46c7-8811-db294e5cab73`::moving from state preparing -> state finished
>Thread-2297::DEBUG::2012-08-30 13:23:16,392::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2297::DEBUG::2012-08-30 13:23:16,392::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2297::DEBUG::2012-08-30 13:23:16,392::task::978::TaskManager.Task::(_decref) Task=`e3c943a9-f675-46c7-8811-db294e5cab73`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:23:18,913::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2299::DEBUG::2012-08-30 13:23:19,553::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2299::DEBUG::2012-08-30 13:23:19,553::task::588::TaskManager.Task::(_updateState) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::moving from state init -> state preparing
>Thread-2299::INFO::2012-08-30 13:23:19,553::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2299::DEBUG::2012-08-30 13:23:19,554::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ab67c026-2633-4ded-8711-861c4eeece2e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2299::DEBUG::2012-08-30 13:23:19,555::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2299::DEBUG::2012-08-30 13:23:19,555::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2299::DEBUG::2012-08-30 13:23:19,555::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`ab67c026-2633-4ded-8711-861c4eeece2e`::Granted request
>Thread-2299::DEBUG::2012-08-30 13:23:19,555::task::817::TaskManager.Task::(resourceAcquired) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2299::DEBUG::2012-08-30 13:23:19,556::task::978::TaskManager.Task::(_decref) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::ref 1 aborting False
>Thread-2299::INFO::2012-08-30 13:23:19,559::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2299::DEBUG::2012-08-30 13:23:19,559::task::1172::TaskManager.Task::(prepare) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2299::DEBUG::2012-08-30 13:23:19,559::task::588::TaskManager.Task::(_updateState) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::moving from state preparing -> state finished
>Thread-2299::DEBUG::2012-08-30 13:23:19,559::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2299::DEBUG::2012-08-30 13:23:19,560::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2299::DEBUG::2012-08-30 13:23:19,561::task::978::TaskManager.Task::(_decref) Task=`762b2ae3-c751-4335-9fe9-03d53097b55a`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:23:20,990::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:23,915::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:23,915::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:25,991::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:23:26,740::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:23:28,916::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::DEBUG::2012-08-30 13:23:30,992::task::588::TaskManager.Task::(_updateState) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::moving from state init -> state preparing
>Thread-104::INFO::2012-08-30 13:23:30,993::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='fadf098f-91fb-4705-a710-afc1b3a96989', volUUID='c8df1315-81fc-423f-890b-eabb0d24c361', options=None)
>Thread-104::DEBUG::2012-08-30 13:23:30,993::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ecbb9aef-8e55-4b0e-be31-90f83d2674a4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-104::DEBUG::2012-08-30 13:23:30,993::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-104::DEBUG::2012-08-30 13:23:30,993::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-104::DEBUG::2012-08-30 13:23:30,994::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`ecbb9aef-8e55-4b0e-be31-90f83d2674a4`::Granted request
>Thread-104::DEBUG::2012-08-30 13:23:30,994::task::817::TaskManager.Task::(resourceAcquired) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-104::DEBUG::2012-08-30 13:23:30,994::task::978::TaskManager.Task::(_decref) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::ref 1 aborting False
>Thread-104::DEBUG::2012-08-30 13:23:30,996::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::DEBUG::2012-08-30 13:23:30,998::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for c8df1315-81fc-423f-890b-eabb0d24c361
>Thread-104::INFO::2012-08-30 13:23:30,999::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:23:30,999::task::1172::TaskManager.Task::(prepare) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::finished: {'truesize': '4096', 'apparentsize': '8589934592'}
>Thread-104::DEBUG::2012-08-30 13:23:30,999::task::588::TaskManager.Task::(_updateState) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::moving from state preparing -> state finished
>Thread-104::DEBUG::2012-08-30 13:23:31,000::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-104::DEBUG::2012-08-30 13:23:31,000::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-104::DEBUG::2012-08-30 13:23:31,000::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-104::DEBUG::2012-08-30 13:23:31,000::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-104::DEBUG::2012-08-30 13:23:31,001::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-104::DEBUG::2012-08-30 13:23:31,001::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-104::DEBUG::2012-08-30 13:23:31,001::task::978::TaskManager.Task::(_decref) Task=`e1d0c357-c364-4fae-a05d-64e486abe170`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:23:31,002::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:31,002::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:31,002::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:31,003::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2303::DEBUG::2012-08-30 13:23:32,724::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2303::DEBUG::2012-08-30 13:23:32,725::task::588::TaskManager.Task::(_updateState) Task=`6f798e71-d106-46db-a85e-60af2ab50a43`::moving from state init -> state preparing
>Thread-2303::INFO::2012-08-30 13:23:32,725::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2303::INFO::2012-08-30 13:23:32,725::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2303::DEBUG::2012-08-30 13:23:32,725::task::1172::TaskManager.Task::(prepare) Task=`6f798e71-d106-46db-a85e-60af2ab50a43`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2303::DEBUG::2012-08-30 13:23:32,726::task::588::TaskManager.Task::(_updateState) Task=`6f798e71-d106-46db-a85e-60af2ab50a43`::moving from state preparing -> state finished
>Thread-2303::DEBUG::2012-08-30 13:23:32,726::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2303::DEBUG::2012-08-30 13:23:32,726::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2303::DEBUG::2012-08-30 13:23:32,726::task::978::TaskManager.Task::(_decref) Task=`6f798e71-d106-46db-a85e-60af2ab50a43`::ref 0 aborting False
>VM Channels Listener::DEBUG::2012-08-30 13:23:33,745::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:23:33,918::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2304::DEBUG::2012-08-30 13:23:35,870::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2304::DEBUG::2012-08-30 13:23:35,871::task::588::TaskManager.Task::(_updateState) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::moving from state init -> state preparing
>Thread-2304::INFO::2012-08-30 13:23:35,871::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2304::DEBUG::2012-08-30 13:23:35,872::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1c36469c-e257-421b-b13a-8a29bf5d6509`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2304::DEBUG::2012-08-30 13:23:35,872::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2304::DEBUG::2012-08-30 13:23:35,872::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2304::DEBUG::2012-08-30 13:23:35,873::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`1c36469c-e257-421b-b13a-8a29bf5d6509`::Granted request
>Thread-2304::DEBUG::2012-08-30 13:23:35,873::task::817::TaskManager.Task::(resourceAcquired) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2304::DEBUG::2012-08-30 13:23:35,873::task::978::TaskManager.Task::(_decref) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::ref 1 aborting False
>Thread-2304::INFO::2012-08-30 13:23:35,876::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2304::DEBUG::2012-08-30 13:23:35,876::task::1172::TaskManager.Task::(prepare) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2304::DEBUG::2012-08-30 13:23:35,877::task::588::TaskManager.Task::(_updateState) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::moving from state preparing -> state finished
>Thread-2304::DEBUG::2012-08-30 13:23:35,877::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2304::DEBUG::2012-08-30 13:23:35,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2304::DEBUG::2012-08-30 13:23:35,877::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2304::DEBUG::2012-08-30 13:23:35,878::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2304::DEBUG::2012-08-30 13:23:35,878::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2304::DEBUG::2012-08-30 13:23:35,878::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2304::DEBUG::2012-08-30 13:23:35,878::task::978::TaskManager.Task::(_decref) Task=`a92f08e1-6d5d-467e-9753-60ab2ce03a26`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:23:36,004::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:38,919::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:38,919::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:41,005::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2306::DEBUG::2012-08-30 13:23:41,027::task::588::TaskManager.Task::(_updateState) Task=`15b626d5-3fd0-462e-9e74-8d411fef50d2`::moving from state init -> state preparing
>Thread-2306::INFO::2012-08-30 13:23:41,028::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2306::INFO::2012-08-30 13:23:41,028::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164413452148', 'lastCheck': 1346304211.6756511, 'code': 0, 'valid': True}}
>Thread-2306::DEBUG::2012-08-30 13:23:41,028::task::1172::TaskManager.Task::(prepare) Task=`15b626d5-3fd0-462e-9e74-8d411fef50d2`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00164413452148', 'lastCheck': 1346304211.6756511, 'code': 0, 'valid': True}}
>Thread-2306::DEBUG::2012-08-30 13:23:41,028::task::588::TaskManager.Task::(_updateState) Task=`15b626d5-3fd0-462e-9e74-8d411fef50d2`::moving from state preparing -> state finished
>Thread-2306::DEBUG::2012-08-30 13:23:41,028::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2306::DEBUG::2012-08-30 13:23:41,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2306::DEBUG::2012-08-30 13:23:41,029::task::978::TaskManager.Task::(_decref) Task=`15b626d5-3fd0-462e-9e74-8d411fef50d2`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:23:43,921::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2307::DEBUG::2012-08-30 13:23:44,176::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2307::DEBUG::2012-08-30 13:23:44,176::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-104::ERROR::2012-08-30 13:23:46,006::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:46,006::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:48,922::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2308::DEBUG::2012-08-30 13:23:49,056::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2308::DEBUG::2012-08-30 13:23:49,057::task::588::TaskManager.Task::(_updateState) Task=`aa4847c9-629a-4a77-b4a5-a91b8ad661aa`::moving from state init -> state preparing
>Thread-2308::INFO::2012-08-30 13:23:49,057::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2308::INFO::2012-08-30 13:23:49,057::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2308::DEBUG::2012-08-30 13:23:49,058::task::1172::TaskManager.Task::(prepare) Task=`aa4847c9-629a-4a77-b4a5-a91b8ad661aa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2308::DEBUG::2012-08-30 13:23:49,058::task::588::TaskManager.Task::(_updateState) Task=`aa4847c9-629a-4a77-b4a5-a91b8ad661aa`::moving from state preparing -> state finished
>Thread-2308::DEBUG::2012-08-30 13:23:49,058::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2308::DEBUG::2012-08-30 13:23:49,059::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2308::DEBUG::2012-08-30 13:23:49,059::task::978::TaskManager.Task::(_decref) Task=`aa4847c9-629a-4a77-b4a5-a91b8ad661aa`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:23:51,007::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2310::DEBUG::2012-08-30 13:23:52,204::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2310::DEBUG::2012-08-30 13:23:52,205::task::588::TaskManager.Task::(_updateState) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::moving from state init -> state preparing
>Thread-2310::INFO::2012-08-30 13:23:52,205::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2310::DEBUG::2012-08-30 13:23:52,205::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`50903081-43f4-4a53-9177-6c080584534c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2310::DEBUG::2012-08-30 13:23:52,205::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2310::DEBUG::2012-08-30 13:23:52,206::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2310::DEBUG::2012-08-30 13:23:52,206::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`50903081-43f4-4a53-9177-6c080584534c`::Granted request
>Thread-2310::DEBUG::2012-08-30 13:23:52,206::task::817::TaskManager.Task::(resourceAcquired) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2310::DEBUG::2012-08-30 13:23:52,207::task::978::TaskManager.Task::(_decref) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::ref 1 aborting False
>Thread-2310::INFO::2012-08-30 13:23:52,209::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2310::DEBUG::2012-08-30 13:23:52,210::task::1172::TaskManager.Task::(prepare) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2310::DEBUG::2012-08-30 13:23:52,210::task::588::TaskManager.Task::(_updateState) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::moving from state preparing -> state finished
>Thread-2310::DEBUG::2012-08-30 13:23:52,210::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2310::DEBUG::2012-08-30 13:23:52,210::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2310::DEBUG::2012-08-30 13:23:52,211::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2310::DEBUG::2012-08-30 13:23:52,211::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2310::DEBUG::2012-08-30 13:23:52,211::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2310::DEBUG::2012-08-30 13:23:52,211::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2310::DEBUG::2012-08-30 13:23:52,212::task::978::TaskManager.Task::(_decref) Task=`624ed996-43a0-4b5e-ab54-9457146e8d26`::ref 0 aborting False
>Thread-40::DEBUG::2012-08-30 13:23:53,923::task::588::TaskManager.Task::(_updateState) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::moving from state init -> state preparing
>Thread-40::INFO::2012-08-30 13:23:53,923::logUtils::37::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='32fcf211-5c90-4362-911c-36bbc97a8a91', spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', imgUUID='5c5dddcd-25c1-47ca-b812-1cc85161bece', volUUID='e8a793c4-36f7-4419-a478-7ac15505e93f', options=None)
>Thread-40::DEBUG::2012-08-30 13:23:53,924::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9d7aabef-71e6-47bb-aea1-402faec64d10`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-40::DEBUG::2012-08-30 13:23:53,924::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' for lock type 'shared'
>Thread-40::DEBUG::2012-08-30 13:23:53,924::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free. Now locking as 'shared' (1 active user)
>Thread-40::DEBUG::2012-08-30 13:23:53,924::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.32fcf211-5c90-4362-911c-36bbc97a8a91`ReqID=`9d7aabef-71e6-47bb-aea1-402faec64d10`::Granted request
>Thread-40::DEBUG::2012-08-30 13:23:53,925::task::817::TaskManager.Task::(resourceAcquired) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::_resourcesAcquired: Storage.32fcf211-5c90-4362-911c-36bbc97a8a91 (shared)
>Thread-40::DEBUG::2012-08-30 13:23:53,925::task::978::TaskManager.Task::(_decref) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::ref 1 aborting False
>Thread-40::DEBUG::2012-08-30 13:23:53,927::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::DEBUG::2012-08-30 13:23:53,929::fileVolume::552::Storage.Volume::(validateVolumePath) validate path for e8a793c4-36f7-4419-a478-7ac15505e93f
>Thread-40::INFO::2012-08-30 13:23:53,930::logUtils::39::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:23:53,930::task::1172::TaskManager.Task::(prepare) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::finished: {'truesize': '2853515264', 'apparentsize': '8589934592'}
>Thread-40::DEBUG::2012-08-30 13:23:53,930::task::588::TaskManager.Task::(_updateState) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::moving from state preparing -> state finished
>Thread-40::DEBUG::2012-08-30 13:23:53,931::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91': < ResourceRef 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', isValid: 'True' obj: 'None'>}
>Thread-40::DEBUG::2012-08-30 13:23:53,931::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-40::DEBUG::2012-08-30 13:23:53,931::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91'
>Thread-40::DEBUG::2012-08-30 13:23:53,931::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' (0 active users)
>Thread-40::DEBUG::2012-08-30 13:23:53,932::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91' is free, finding out if anyone is waiting for it.
>Thread-40::DEBUG::2012-08-30 13:23:53,932::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.32fcf211-5c90-4362-911c-36bbc97a8a91', Clearing records.
>Thread-40::DEBUG::2012-08-30 13:23:53,932::task::978::TaskManager.Task::(_decref) Task=`2da7806e-ed48-48f6-a524-f2f43908612a`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:23:53,933::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:53,933::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x28a7bc8>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 122, in _sampleDisk
>    diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1746, in blockStats
>    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:53,933::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x28ab8f0>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 163, in _sampleDiskLatency
>    libvirt_qemu.VIR_DOMAIN_QEMU_MONITOR_COMMAND_DEFAULT)
>  File "/usr/lib64/python2.6/site-packages/libvirt_qemu.py", line 37, in qemuMonitorCommand
>    if ret is None: raise libvirt.libvirtError('virDomainQemuMonitorCommand() failed')
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:23:53,934::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:23:56,008::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:23:56,764::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 60.
>Thread-40::ERROR::2012-08-30 13:23:58,935::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:24:01,009::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:24:01,010::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>VM Channels Listener::DEBUG::2012-08-30 13:24:03,770::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 63.
>Thread-40::ERROR::2012-08-30 13:24:03,936::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2314::DEBUG::2012-08-30 13:24:05,390::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2314::DEBUG::2012-08-30 13:24:05,391::task::588::TaskManager.Task::(_updateState) Task=`794f26d8-b8e7-49ac-a808-89b26836eb9f`::moving from state init -> state preparing
>Thread-2314::INFO::2012-08-30 13:24:05,391::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2314::INFO::2012-08-30 13:24:05,391::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2314::DEBUG::2012-08-30 13:24:05,392::task::1172::TaskManager.Task::(prepare) Task=`794f26d8-b8e7-49ac-a808-89b26836eb9f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 3}}
>Thread-2314::DEBUG::2012-08-30 13:24:05,392::task::588::TaskManager.Task::(_updateState) Task=`794f26d8-b8e7-49ac-a808-89b26836eb9f`::moving from state preparing -> state finished
>Thread-2314::DEBUG::2012-08-30 13:24:05,392::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2314::DEBUG::2012-08-30 13:24:05,392::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2314::DEBUG::2012-08-30 13:24:05,393::task::978::TaskManager.Task::(_decref) Task=`794f26d8-b8e7-49ac-a808-89b26836eb9f`::ref 0 aborting False
>Thread-104::ERROR::2012-08-30 13:24:06,010::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2315::DEBUG::2012-08-30 13:24:08,540::BindingXMLRPC::164::vds::(wrapper) [10.66.5.12]
>Thread-2316::DEBUG::2012-08-30 13:24:08,540::task::588::TaskManager.Task::(_updateState) Task=`b3a65101-83a8-47dc-ad17-6d460014534d`::moving from state init -> state preparing
>Thread-2315::DEBUG::2012-08-30 13:24:08,541::task::588::TaskManager.Task::(_updateState) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::moving from state init -> state preparing
>Thread-2316::INFO::2012-08-30 13:24:08,541::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-2315::INFO::2012-08-30 13:24:08,541::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='b25c9a7d-4634-406d-9881-bab3e1793946', options=None)
>Thread-2316::INFO::2012-08-30 13:24:08,541::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178599357605', 'lastCheck': 1346304241.6961839, 'code': 0, 'valid': True}}
>Thread-2315::DEBUG::2012-08-30 13:24:08,542::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2cff57be-a476-488b-8120-2acf937cb3e1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'
>Thread-2316::DEBUG::2012-08-30 13:24:08,542::task::1172::TaskManager.Task::(prepare) Task=`b3a65101-83a8-47dc-ad17-6d460014534d`::finished: {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'delay': '0.00178599357605', 'lastCheck': 1346304241.6961839, 'code': 0, 'valid': True}}
>Thread-2315::DEBUG::2012-08-30 13:24:08,542::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' for lock type 'shared'
>Thread-2316::DEBUG::2012-08-30 13:24:08,542::task::588::TaskManager.Task::(_updateState) Task=`b3a65101-83a8-47dc-ad17-6d460014534d`::moving from state preparing -> state finished
>Thread-2315::DEBUG::2012-08-30 13:24:08,543::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free. Now locking as 'shared' (1 active user)
>Thread-2316::DEBUG::2012-08-30 13:24:08,543::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-2315::DEBUG::2012-08-30 13:24:08,543::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.b25c9a7d-4634-406d-9881-bab3e1793946`ReqID=`2cff57be-a476-488b-8120-2acf937cb3e1`::Granted request
>Thread-2316::DEBUG::2012-08-30 13:24:08,543::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2315::DEBUG::2012-08-30 13:24:08,543::task::817::TaskManager.Task::(resourceAcquired) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::_resourcesAcquired: Storage.b25c9a7d-4634-406d-9881-bab3e1793946 (shared)
>Thread-2316::DEBUG::2012-08-30 13:24:08,544::task::978::TaskManager.Task::(_decref) Task=`b3a65101-83a8-47dc-ad17-6d460014534d`::ref 0 aborting False
>Thread-2315::DEBUG::2012-08-30 13:24:08,544::task::978::TaskManager.Task::(_decref) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::ref 1 aborting False
>Thread-2315::INFO::2012-08-30 13:24:08,549::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2315::DEBUG::2012-08-30 13:24:08,549::task::1172::TaskManager.Task::(prepare) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::finished: {'info': {'spm_id': 2, 'master_uuid': '32fcf211-5c90-4362-911c-36bbc97a8a91', 'name': 'whuang2', 'version': '3', 'domains': '32fcf211-5c90-4362-911c-36bbc97a8a91:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 3}, 'dominfo': {'32fcf211-5c90-4362-911c-36bbc97a8a91': {'status': 'Active', 'diskfree': '57070845952', 'alerts': [], 'disktotal': '152358092800'}}}
>Thread-2315::DEBUG::2012-08-30 13:24:08,549::task::588::TaskManager.Task::(_updateState) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::moving from state preparing -> state finished
>Thread-2315::DEBUG::2012-08-30 13:24:08,549::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b25c9a7d-4634-406d-9881-bab3e1793946': < ResourceRef 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', isValid: 'True' obj: 'None'>}
>Thread-2315::DEBUG::2012-08-30 13:24:08,550::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-2315::DEBUG::2012-08-30 13:24:08,550::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946'
>Thread-2315::DEBUG::2012-08-30 13:24:08,550::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' (0 active users)
>Thread-2315::DEBUG::2012-08-30 13:24:08,550::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946' is free, finding out if anyone is waiting for it.
>Thread-2315::DEBUG::2012-08-30 13:24:08,551::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b25c9a7d-4634-406d-9881-bab3e1793946', Clearing records.
>Thread-2315::DEBUG::2012-08-30 13:24:08,551::task::978::TaskManager.Task::(_decref) Task=`e06a8e77-0cce-4574-8bcd-3a9ab7752e76`::ref 0 aborting False
>Thread-40::ERROR::2012-08-30 13:24:08,937::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x28a7b58>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 112, in _sampleCpu
>    state, maxMem, memory, nrVirtCpu, cpuTime = self._vm._dom.info()
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1781, in info
>    if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-40::ERROR::2012-08-30 13:24:08,938::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-104::ERROR::2012-08-30 13:24:11,011::utils::399::vm.Vm::(collect) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed
>Thread-2317::DEBUG::2012-08-30 13:24:11,410::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Disk hdc stats not available
>Thread-2317::DEBUG::2012-08-30 13:24:11,410::libvirtvm::243::vm.Vm::(_getDiskStats) vmId=`8554f46f-d84e-4486-ba34-047f88dfdf74`::Disk hdc stats not available
>Thread-40::ERROR::2012-08-30 13:24:13,939::utils::399::vm.Vm::(collect) vmId=`07e35260-cae0-4000-8ea5-37528f010744`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x28a7c38>
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 395, in collect
>    statsFunction()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 272, in __call__
>    retValue = self._function(*args, **kwargs)
>  File "/usr/share/vdsm/libvirtvm.py", line 182, in _sampleNet
>    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
>  File "/usr/share/vdsm/libvirtvm.py", line 494, in f
>    ret = attr(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper
>    ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1793, in interfaceStats
>    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
>libvirtError: internal error client socket is closed

Actions: View

Attachments on bug 852956: 608047 | 608048 | 736326 | 741901