Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 853936 Details for
Bug 1056649
[hosted-engine] When hosted engine's console is set to SPICE then it fails
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
vdsm.log
vdsm.log (text/x-log), 593.34 KB, created by
Jiri Belka
on 2014-01-22 15:59:21 UTC
(
hide
)
Description:
vdsm.log
Filename:
MIME Type:
Creator:
Jiri Belka
Created:
2014-01-22 15:59:21 UTC
Size:
593.34 KB
patch
obsolete
>MainThread::INFO::2014-01-22 16:32:38,828::vdsm::101::vds::(run) (PID: 3414) I am the actual vdsm 4.13.2-0.6.el6ev dell-r210ii-13.rhev.lab.eng.brq.redhat.com (2.6.32-431.el6.x86_64) >MainThread::DEBUG::2014-01-22 16:32:38,993::resourceManager::420::ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2014-01-22 16:32:38,994::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 >MainThread::DEBUG::2014-01-22 16:32:39,342::multipath::153::Storage.Multipath::(isEnabled) multipath Defaulting to False >MainThread::DEBUG::2014-01-22 16:32:39,343::multipath::171::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /bin/cp /tmp/tmprI0APe /etc/multipath.conf' (cwd None) >MainThread::DEBUG::2014-01-22 16:32:39,398::multipath::171::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-01-22 16:32:39,398::multipath::177::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /sbin/multipath -F' (cwd None) >MainThread::DEBUG::2014-01-22 16:32:39,409::multipath::177::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-01-22 16:32:39,410::multipath::180::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /usr/bin/vdsm-tool service-reload multipathd' (cwd None) >MainThread::DEBUG::2014-01-22 16:32:39,542::multipath::180::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-01-22 16:32:39,542::hsm::427::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2014-01-22 16:32:39,677::hsm::427::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-01-22 16:32:39,678::hsm::451::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >MainThread::DEBUG::2014-01-22 16:32:39,682::hsm::483::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >MainThread::DEBUG::2014-01-22 16:32:39,683::hsm::484::Storage.HSM::(__cleanStorageRepository) Mount list: [] >MainThread::DEBUG::2014-01-22 16:32:39,683::hsm::486::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >MainThread::DEBUG::2014-01-22 16:32:39,685::hsm::529::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >MainThread::INFO::2014-01-22 16:32:39,687::dispatcher::98::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >storageRefresh::DEBUG::2014-01-22 16:32:39,687::lvm::338::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >MainThread::DEBUG::2014-01-22 16:32:39,688::task::579::TaskManager.Task::(_updateState) Task=`c048480a-3f20-4a46-93cb-0b4cdbe851f9`::moving from state init -> state preparing >MainThread::INFO::2014-01-22 16:32:39,689::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x7f71f6ecb4d0>>) >MainThread::INFO::2014-01-22 16:32:39,689::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None >MainThread::DEBUG::2014-01-22 16:32:39,690::task::1168::TaskManager.Task::(prepare) Task=`c048480a-3f20-4a46-93cb-0b4cdbe851f9`::finished: None >MainThread::DEBUG::2014-01-22 16:32:39,690::task::579::TaskManager.Task::(_updateState) Task=`c048480a-3f20-4a46-93cb-0b4cdbe851f9`::moving from state preparing -> state finished >MainThread::DEBUG::2014-01-22 16:32:39,690::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2014-01-22 16:32:39,690::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2014-01-22 16:32:39,691::task::974::TaskManager.Task::(_decref) Task=`c048480a-3f20-4a46-93cb-0b4cdbe851f9`::ref 0 aborting False >storageRefresh::DEBUG::2014-01-22 16:32:39,693::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >MainThread::INFO::2014-01-22 16:32:39,700::momIF::47::MOM::(__init__) Starting up MOM >clientIFinit::DEBUG::2014-01-22 16:32:39,702::libvirtconnection::131::libvirtconnection::(get) trying to connect libvirt >MainThread::INFO::2014-01-22 16:32:39,702::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds. >VM Channels Listener::INFO::2014-01-22 16:32:39,709::vmChannels::170::vds::(run) Starting VM channels listener thread. >storageRefresh::DEBUG::2014-01-22 16:32:39,802::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:32:39,804::lvm::363::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:39,805::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:39,808::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-13::DEBUG::2014-01-22 16:32:39,873::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call getHardwareInfo with () {} >Thread-13::DEBUG::2014-01-22 16:32:39,874::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >storageRefresh::DEBUG::2014-01-22 16:32:39,892::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:32:39,892::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:39,893::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >Thread-13::DEBUG::2014-01-22 16:32:39,932::BindingXMLRPC::991::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210 II', 'systemSerialNumber': '34SRJ5J', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0034-5310-8052-B3C04F4A354A', 'systemManufacturer': 'Dell Inc.'}} >storageRefresh::DEBUG::2014-01-22 16:32:39,988::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:32:39,989::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:39,989::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-14::DEBUG::2014-01-22 16:32:40,002::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-14::DEBUG::2014-01-22 16:32:40,004::task::579::TaskManager.Task::(_updateState) Task=`84065248-529a-4057-b2ca-d84373ced252`::moving from state init -> state preparing >Thread-14::INFO::2014-01-22 16:32:40,004::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >Thread-14::INFO::2014-01-22 16:32:40,005::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >Thread-14::DEBUG::2014-01-22 16:32:40,005::task::1168::TaskManager.Task::(prepare) Task=`84065248-529a-4057-b2ca-d84373ced252`::finished: {'poollist': []} >Thread-14::DEBUG::2014-01-22 16:32:40,005::task::579::TaskManager.Task::(_updateState) Task=`84065248-529a-4057-b2ca-d84373ced252`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-01-22 16:32:40,006::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-01-22 16:32:40,006::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-01-22 16:32:40,007::task::974::TaskManager.Task::(_decref) Task=`84065248-529a-4057-b2ca-d84373ced252`::ref 0 aborting False >storageRefresh::DEBUG::2014-01-22 16:32:40,078::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:32:40,078::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:40,079::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >storageRefresh::DEBUG::2014-01-22 16:32:40,079::misc::811::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-01-22 16:32:40,079::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >storageRefresh::DEBUG::2014-01-22 16:32:40,079::misc::811::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-01-22 16:32:40,080::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:32:40,094::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >storageRefresh::DEBUG::2014-01-22 16:32:40,094::misc::819::SamplingMethod::(__call__) Returning last result >storageRefresh::DEBUG::2014-01-22 16:32:43,136::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:32:43,218::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:32:43,218::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:43,221::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:43,222::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:43,222::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:43,222::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:43,222::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:32:43,222::misc::819::SamplingMethod::(__call__) Returning last result >Thread-15::DEBUG::2014-01-22 16:34:55,134::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-15::DEBUG::2014-01-22 16:34:55,135::task::579::TaskManager.Task::(_updateState) Task=`e82cd88c-47aa-47b0-b9a8-c1d186bbc270`::moving from state init -> state preparing >Thread-15::INFO::2014-01-22 16:34:55,135::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': '10.34.63.199:/jbelka/hosted', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}], options=None) >Thread-15::DEBUG::2014-01-22 16:34:55,139::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.63.199:/jbelka/hosted /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted' (cwd None) >Thread-15::DEBUG::2014-01-22 16:34:55,262::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted >Thread-15::DEBUG::2014-01-22 16:34:55,265::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: () >Thread-15::DEBUG::2014-01-22 16:34:55,265::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-15::INFO::2014-01-22 16:34:55,265::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}]} >Thread-15::DEBUG::2014-01-22 16:34:55,265::task::1168::TaskManager.Task::(prepare) Task=`e82cd88c-47aa-47b0-b9a8-c1d186bbc270`::finished: {'statuslist': [{'status': 0, 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}]} >Thread-15::DEBUG::2014-01-22 16:34:55,266::task::579::TaskManager.Task::(_updateState) Task=`e82cd88c-47aa-47b0-b9a8-c1d186bbc270`::moving from state preparing -> state finished >Thread-15::DEBUG::2014-01-22 16:34:55,266::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15::DEBUG::2014-01-22 16:34:55,266::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15::DEBUG::2014-01-22 16:34:55,266::task::974::TaskManager.Task::(_decref) Task=`e82cd88c-47aa-47b0-b9a8-c1d186bbc270`::ref 0 aborting False >Thread-16::DEBUG::2014-01-22 16:34:55,323::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-16::DEBUG::2014-01-22 16:34:55,323::task::579::TaskManager.Task::(_updateState) Task=`55be6863-b117-4a29-b86d-d8ad94a460d2`::moving from state init -> state preparing >Thread-16::INFO::2014-01-22 16:34:55,324::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainsList(spUUID='00000000-0000-0000-0000-000000000000', domainClass=None, storageType=None, remotePath=None, options=None) >Thread-16::DEBUG::2014-01-22 16:34:55,324::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-16::DEBUG::2014-01-22 16:34:55,324::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-16::DEBUG::2014-01-22 16:34:55,324::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-16::DEBUG::2014-01-22 16:34:55,325::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-16::DEBUG::2014-01-22 16:34:55,326::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-16::DEBUG::2014-01-22 16:34:55,347::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-16::DEBUG::2014-01-22 16:34:55,348::misc::819::SamplingMethod::(__call__) Returning last result >Thread-16::DEBUG::2014-01-22 16:34:58,382::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-16::DEBUG::2014-01-22 16:34:58,472::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-16::DEBUG::2014-01-22 16:34:58,473::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:34:58,473::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:34:58,474::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:34:58,474::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:34:58,474::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:34:58,474::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:34:58,475::misc::819::SamplingMethod::(__call__) Returning last result >Thread-16::DEBUG::2014-01-22 16:34:58,475::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:34:58,477::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-16::DEBUG::2014-01-22 16:34:58,589::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >Thread-16::DEBUG::2014-01-22 16:34:58,590::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16::INFO::2014-01-22 16:34:58,642::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainsList, Return response: {'domlist': []} >Thread-16::DEBUG::2014-01-22 16:34:58,643::task::1168::TaskManager.Task::(prepare) Task=`55be6863-b117-4a29-b86d-d8ad94a460d2`::finished: {'domlist': []} >Thread-16::DEBUG::2014-01-22 16:34:58,643::task::579::TaskManager.Task::(_updateState) Task=`55be6863-b117-4a29-b86d-d8ad94a460d2`::moving from state preparing -> state finished >Thread-16::DEBUG::2014-01-22 16:34:58,643::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-16::DEBUG::2014-01-22 16:34:58,644::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-16::DEBUG::2014-01-22 16:34:58,644::task::974::TaskManager.Task::(_decref) Task=`55be6863-b117-4a29-b86d-d8ad94a460d2`::ref 0 aborting False >Thread-18::DEBUG::2014-01-22 16:34:58,705::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-18::DEBUG::2014-01-22 16:34:58,705::task::579::TaskManager.Task::(_updateState) Task=`d5f562d5-4f9f-4553-803f-402b82c19a23`::moving from state init -> state preparing >Thread-18::INFO::2014-01-22 16:34:58,706::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': '10.34.63.199:/jbelka/hosted', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}], options=None) >Thread-18::DEBUG::2014-01-22 16:34:58,706::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted' (cwd None) >Thread-18::DEBUG::2014-01-22 16:34:58,739::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-18::DEBUG::2014-01-22 16:34:58,740::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-18::DEBUG::2014-01-22 16:34:58,740::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-18::DEBUG::2014-01-22 16:34:58,740::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-18::DEBUG::2014-01-22 16:34:58,740::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-18::DEBUG::2014-01-22 16:34:58,753::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-18::DEBUG::2014-01-22 16:34:58,753::misc::819::SamplingMethod::(__call__) Returning last result >Thread-18::DEBUG::2014-01-22 16:35:00,791::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-18::DEBUG::2014-01-22 16:35:00,874::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-18::DEBUG::2014-01-22 16:35:00,874::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2014-01-22 16:35:00,875::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2014-01-22 16:35:00,875::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2014-01-22 16:35:00,875::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2014-01-22 16:35:00,876::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2014-01-22 16:35:00,876::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2014-01-22 16:35:00,876::misc::819::SamplingMethod::(__call__) Returning last result >Thread-18::INFO::2014-01-22 16:35:00,876::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}]} >Thread-18::DEBUG::2014-01-22 16:35:00,877::task::1168::TaskManager.Task::(prepare) Task=`d5f562d5-4f9f-4553-803f-402b82c19a23`::finished: {'statuslist': [{'status': 0, 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}]} >Thread-18::DEBUG::2014-01-22 16:35:00,877::task::579::TaskManager.Task::(_updateState) Task=`d5f562d5-4f9f-4553-803f-402b82c19a23`::moving from state preparing -> state finished >Thread-18::DEBUG::2014-01-22 16:35:00,877::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-18::DEBUG::2014-01-22 16:35:00,877::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-18::DEBUG::2014-01-22 16:35:00,877::task::974::TaskManager.Task::(_decref) Task=`d5f562d5-4f9f-4553-803f-402b82c19a23`::ref 0 aborting False >MainThread::DEBUG::2014-01-22 16:37:43,545::vdsm::45::vds::(sigtermHandler) Received signal 15 >MainThread::INFO::2014-01-22 16:37:43,845::vmChannels::183::vds::(stop) VM channels listener was stopped. >MainThread::INFO::2014-01-22 16:37:43,845::momIF::84::MOM::(stop) Shutting down MOM >MainThread::DEBUG::2014-01-22 16:37:43,846::task::579::TaskManager.Task::(_updateState) Task=`a0b27721-b782-4400-830d-b4bd86f1d288`::moving from state init -> state preparing >MainThread::INFO::2014-01-22 16:37:43,847::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) >Thread-11::DEBUG::2014-01-22 16:37:43,848::storageServer::698::ConnectionMonitor::(_monitorConnections) Monitoring stopped >VM Channels Listener::INFO::2014-01-22 16:37:44,044::vmChannels::178::vds::(run) VM channels listener thread has ended. >MainThread::INFO::2014-01-22 16:37:44,391::domainMonitor::125::Storage.DomainMonitor::(close) Stopping domain monitors >MainThread::DEBUG::2014-01-22 16:37:44,391::taskManager::80::TaskManager::(prepareForShutdown) Request to stop all tasks >MainThread::INFO::2014-01-22 16:37:44,391::logUtils::47::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None >MainThread::DEBUG::2014-01-22 16:37:44,391::task::1168::TaskManager.Task::(prepare) Task=`a0b27721-b782-4400-830d-b4bd86f1d288`::finished: None >MainThread::DEBUG::2014-01-22 16:37:44,391::task::579::TaskManager.Task::(_updateState) Task=`a0b27721-b782-4400-830d-b4bd86f1d288`::moving from state preparing -> state finished >MainThread::DEBUG::2014-01-22 16:37:44,391::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2014-01-22 16:37:44,391::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2014-01-22 16:37:44,391::task::974::TaskManager.Task::(_decref) Task=`a0b27721-b782-4400-830d-b4bd86f1d288`::ref 0 aborting False >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::107::vds::(run) VDSM main thread ended. Waiting for 15 other threads... >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <_MainThread(MainThread, started 140127484241664)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <Thread(libvirtEventLoop, started daemon 140127349143296)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <PolicyEngine(PolicyEngine, started daemon 140125791057664)> >MainThread::INFO::2014-01-22 16:37:44,392::momIF::84::MOM::(stop) Shutting down MOM >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <MomThread(MOM, started 140126260819712)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <WorkerThread(Thread-6, started daemon 140127091275520)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <WorkerThread(Thread-3, started daemon 140127122745088)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <WorkerThread(Thread-5, started daemon 140127101765376)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <WorkerThread(Thread-9, started daemon 140126720091904)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <WorkerThread(Thread-8, started daemon 140126730581760)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <GuestManager(GuestManager, started daemon 140126218860288)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <WorkerThread(Thread-1, started daemon 140127220815616)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <WorkerThread(Thread-2, started daemon 140127133234944)> >MainThread::INFO::2014-01-22 16:37:44,392::vdsm::111::vds::(run) <WorkerThread(Thread-10, started daemon 140126709602048)> >MainThread::INFO::2014-01-22 16:37:44,393::vdsm::111::vds::(run) <WorkerThread(Thread-7, started daemon 140127080785664)> >MainThread::INFO::2014-01-22 16:37:44,393::vdsm::111::vds::(run) <HostMonitor(HostMonitor, started daemon 140126250329856)> >MainThread::INFO::2014-01-22 16:37:44,393::vdsm::111::vds::(run) <WorkerThread(Thread-4, started daemon 140127112255232)> >MainThread::INFO::2014-01-22 16:37:47,386::vdsm::101::vds::(run) (PID: 4039) I am the actual vdsm 4.13.2-0.6.el6ev dell-r210ii-13.rhev.lab.eng.brq.redhat.com (2.6.32-431.el6.x86_64) >MainThread::DEBUG::2014-01-22 16:37:47,549::resourceManager::420::ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2014-01-22 16:37:47,549::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 >MainThread::WARNING::2014-01-22 16:37:47,552::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists >MainThread::DEBUG::2014-01-22 16:37:47,866::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >MainThread::DEBUG::2014-01-22 16:37:47,883::multipath::144::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving >MainThread::DEBUG::2014-01-22 16:37:47,883::hsm::427::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2014-01-22 16:37:47,994::hsm::427::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-01-22 16:37:47,995::hsm::451::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >MainThread::DEBUG::2014-01-22 16:37:47,999::hsm::483::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >MainThread::DEBUG::2014-01-22 16:37:47,999::hsm::484::Storage.HSM::(__cleanStorageRepository) Mount list: [] >MainThread::DEBUG::2014-01-22 16:37:48,000::hsm::486::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >MainThread::DEBUG::2014-01-22 16:37:48,001::hsm::529::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >MainThread::INFO::2014-01-22 16:37:48,004::dispatcher::98::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >storageRefresh::DEBUG::2014-01-22 16:37:48,004::lvm::338::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >MainThread::DEBUG::2014-01-22 16:37:48,005::task::579::TaskManager.Task::(_updateState) Task=`eb54728e-a042-414f-a9e9-414dca8fa38c`::moving from state init -> state preparing >MainThread::INFO::2014-01-22 16:37:48,006::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x7fe9d84b44d0>>) >MainThread::INFO::2014-01-22 16:37:48,006::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None >MainThread::DEBUG::2014-01-22 16:37:48,006::task::1168::TaskManager.Task::(prepare) Task=`eb54728e-a042-414f-a9e9-414dca8fa38c`::finished: None >MainThread::DEBUG::2014-01-22 16:37:48,007::task::579::TaskManager.Task::(_updateState) Task=`eb54728e-a042-414f-a9e9-414dca8fa38c`::moving from state preparing -> state finished >MainThread::DEBUG::2014-01-22 16:37:48,007::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2014-01-22 16:37:48,007::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2014-01-22 16:37:48,007::task::974::TaskManager.Task::(_decref) Task=`eb54728e-a042-414f-a9e9-414dca8fa38c`::ref 0 aborting False >storageRefresh::DEBUG::2014-01-22 16:37:48,009::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >MainThread::INFO::2014-01-22 16:37:48,019::momIF::47::MOM::(__init__) Starting up MOM >MainThread::INFO::2014-01-22 16:37:48,020::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds. >clientIFinit::DEBUG::2014-01-22 16:37:48,021::libvirtconnection::131::libvirtconnection::(get) trying to connect libvirt >VM Channels Listener::INFO::2014-01-22 16:37:48,023::vmChannels::170::vds::(run) Starting VM channels listener thread. >storageRefresh::DEBUG::2014-01-22 16:37:48,118::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:37:48,119::lvm::363::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:48,120::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:48,123::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:37:48,208::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:37:48,208::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:48,209::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:37:48,297::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:37:48,297::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:48,298::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:37:48,396::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:37:48,397::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:48,397::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >storageRefresh::DEBUG::2014-01-22 16:37:48,397::misc::811::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-01-22 16:37:48,397::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >storageRefresh::DEBUG::2014-01-22 16:37:48,398::misc::811::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-01-22 16:37:48,398::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:37:48,412::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >storageRefresh::DEBUG::2014-01-22 16:37:48,413::misc::819::SamplingMethod::(__call__) Returning last result >Thread-13::DEBUG::2014-01-22 16:37:48,424::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call getHardwareInfo with () {} >Thread-13::DEBUG::2014-01-22 16:37:48,434::BindingXMLRPC::991::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210 II', 'systemSerialNumber': '34SRJ5J', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0034-5310-8052-B3C04F4A354A', 'systemManufacturer': 'Dell Inc.'}} >Thread-14::DEBUG::2014-01-22 16:37:48,491::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call getCapabilities with () {} >Thread-14::DEBUG::2014-01-22 16:37:48,570::utils::489::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-14::DEBUG::2014-01-22 16:37:48,590::utils::509::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-01-22 16:37:48,632::caps::347::root::(_getKeyPackages) rpm package qemu-kvm-rhev not found >Thread-14::DEBUG::2014-01-22 16:37:48,634::caps::347::root::(_getKeyPackages) rpm package qemu-img-rhev not found >Thread-14::DEBUG::2014-01-22 16:37:48,643::BindingXMLRPC::991::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2d2683dd378'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.el6.x86_64', 'buildtime': 1384118394.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.6.el6ev', 'buildtime': 1389025732L, 'version': '4.13.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'libvirt': {'release': '29.el6_5.2', 'buildtime': 1386770011L, 'version': '0.10.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz', 'hooks': {'before_vm_start': {'50_hostedengine': {'md5': '45dde62155b5412eafbfff5ef265acc2'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {}, 'bridges': {}, 'uuid': '4C4C4544-0034-5310-8052-B3C04F4A354A_d4:ae:52:c7:1a:39', 'lastClientIface': 'lo', 'nics': {'em1': {'netmask': '255.255.252.0', 'addr': '10.34.62.205', 'hwaddr': 'd4:ae:52:c7:1a:39', 'cfg': {'IPV6INIT': 'no', 'NM_CONTROLLED': 'yes', 'MTU': '1500', 'HWADDR': 'D4:AE:52:C7:1A:39', 'BOOTPROTO': 'dhcp', 'DEVICE': 'em1', 'TYPE': 'Ethernet', 'ONBOOT': 'yes', 'UUID': '1936afd8-1226-47a2-b0a3-1943b3c42243'}, 'ipv6addrs': [], 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'd4:ae:52:c7:1a:3a', 'cfg': {'UUID': '5fde3cef-9c45-4f65-99c1-7e1441382dab', 'NM_CONTROLLED': 'yes', 'HWADDR': 'D4:AE:52:C7:1A:3A', 'BOOTPROTO': 'dhcp', 'DEVICE': 'em2', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.6', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2d2683dd378', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '7906', 'cpuSpeed': '1600.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '127.0.0.1'}} >Thread-15::DEBUG::2014-01-22 16:37:48,796::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call addNetwork with ('rhevm', '', '', ['em1'], {'blockingdhcp': 'true', 'bootproto': 'dhcp', 'force': 'False', 'ONBOOT': 'yes', 'bridged': 'True'}) {} >Thread-15::WARNING::2014-01-22 16:37:48,796::API::1530::root::(translateNetOptionsToNew) options ONBOOT is deprecated. Use onboot instead >Thread-15::DEBUG::2014-01-22 16:37:49,563::BindingXMLRPC::991::vds::(wrapper) return addNetwork with {'status': {'message': 'Done', 'code': 0}} >storageRefresh::DEBUG::2014-01-22 16:37:53,476::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:37:53,605::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:37:53,605::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:53,606::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:53,606::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:53,606::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:53,607::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:53,607::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:37:53,607::misc::819::SamplingMethod::(__call__) Returning last result >storageRefresh::WARNING::2014-01-22 16:37:53,608::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists >Thread-16::DEBUG::2014-01-22 16:38:02,400::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-16::DEBUG::2014-01-22 16:38:02,406::task::579::TaskManager.Task::(_updateState) Task=`72b1092d-df92-4563-8cd1-d72e03fcad7d`::moving from state init -> state preparing >Thread-16::INFO::2014-01-22 16:38:02,407::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >Thread-16::INFO::2014-01-22 16:38:02,407::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >Thread-16::DEBUG::2014-01-22 16:38:02,407::task::1168::TaskManager.Task::(prepare) Task=`72b1092d-df92-4563-8cd1-d72e03fcad7d`::finished: {'poollist': []} >Thread-16::DEBUG::2014-01-22 16:38:02,407::task::579::TaskManager.Task::(_updateState) Task=`72b1092d-df92-4563-8cd1-d72e03fcad7d`::moving from state preparing -> state finished >Thread-16::DEBUG::2014-01-22 16:38:02,407::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-16::DEBUG::2014-01-22 16:38:02,408::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-16::DEBUG::2014-01-22 16:38:02,408::task::974::TaskManager.Task::(_decref) Task=`72b1092d-df92-4563-8cd1-d72e03fcad7d`::ref 0 aborting False >Thread-17::DEBUG::2014-01-22 16:38:02,460::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-17::DEBUG::2014-01-22 16:38:02,462::task::579::TaskManager.Task::(_updateState) Task=`97509f25-831f-4dfa-89e4-e8ddf18d0507`::moving from state init -> state preparing >Thread-17::INFO::2014-01-22 16:38:02,462::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': '10.34.63.199:/jbelka/hosted', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}], options=None) >Thread-17::DEBUG::2014-01-22 16:38:02,467::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.63.199:/jbelka/hosted /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted' (cwd None) >Thread-17::DEBUG::2014-01-22 16:38:02,591::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted >Thread-17::DEBUG::2014-01-22 16:38:02,594::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: () >Thread-17::DEBUG::2014-01-22 16:38:02,594::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-17::INFO::2014-01-22 16:38:02,594::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}]} >Thread-17::DEBUG::2014-01-22 16:38:02,594::task::1168::TaskManager.Task::(prepare) Task=`97509f25-831f-4dfa-89e4-e8ddf18d0507`::finished: {'statuslist': [{'status': 0, 'id': '051eed1e-cb79-4c11-a7a0-be8538b6eb9c'}]} >Thread-17::DEBUG::2014-01-22 16:38:02,595::task::579::TaskManager.Task::(_updateState) Task=`97509f25-831f-4dfa-89e4-e8ddf18d0507`::moving from state preparing -> state finished >Thread-17::DEBUG::2014-01-22 16:38:02,595::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-17::DEBUG::2014-01-22 16:38:02,595::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-17::DEBUG::2014-01-22 16:38:02,595::task::974::TaskManager.Task::(_decref) Task=`97509f25-831f-4dfa-89e4-e8ddf18d0507`::ref 0 aborting False >Thread-18::DEBUG::2014-01-22 16:38:02,653::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-18::DEBUG::2014-01-22 16:38:02,654::task::579::TaskManager.Task::(_updateState) Task=`0e4b0cc9-5b42-4f2e-a93c-b71b9d4c15a1`::moving from state init -> state preparing >Thread-18::INFO::2014-01-22 16:38:02,654::logUtils::44::dispatcher::(wrapper) Run and protect: createStorageDomain(storageType=1, sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', domainName='hosted_storage', typeSpecificArg='10.34.63.199:/jbelka/hosted', domClass=1, domVersion=3, options=None) >Thread-18::DEBUG::2014-01-22 16:38:02,655::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-18::DEBUG::2014-01-22 16:38:02,655::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-18::DEBUG::2014-01-22 16:38:02,655::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-18::DEBUG::2014-01-22 16:38:02,655::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-18::DEBUG::2014-01-22 16:38:02,655::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-18::DEBUG::2014-01-22 16:38:02,685::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-18::DEBUG::2014-01-22 16:38:02,685::misc::819::SamplingMethod::(__call__) Returning last result >Thread-18::DEBUG::2014-01-22 16:38:05,742::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-18::DEBUG::2014-01-22 16:38:05,881::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-18::DEBUG::2014-01-22 16:38:05,882::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2014-01-22 16:38:05,882::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2014-01-22 16:38:05,883::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2014-01-22 16:38:05,883::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2014-01-22 16:38:05,883::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2014-01-22 16:38:05,884::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2014-01-22 16:38:05,884::misc::819::SamplingMethod::(__call__) Returning last result >Thread-18::ERROR::2014-01-22 16:38:05,884::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-18::ERROR::2014-01-22 16:38:05,884::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-18::DEBUG::2014-01-22 16:38:05,885::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-18::DEBUG::2014-01-22 16:38:05,887::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3900e952-713e-4ea9-b32c-097cd4994bc5' (cwd None) >Thread-18::DEBUG::2014-01-22 16:38:06,012::lvm::309::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3900e952-713e-4ea9-b32c-097cd4994bc5" not found\n Skipping volume group 3900e952-713e-4ea9-b32c-097cd4994bc5\n'; <rc> = 5 >Thread-18::WARNING::2014-01-22 16:38:06,014::lvm::391::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3900e952-713e-4ea9-b32c-097cd4994bc5" not found', ' Skipping volume group 3900e952-713e-4ea9-b32c-097cd4994bc5'] >Thread-18::DEBUG::2014-01-22 16:38:06,014::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-18::ERROR::2014-01-22 16:38:06,071::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3900e952-713e-4ea9-b32c-097cd4994bc5 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3900e952-713e-4ea9-b32c-097cd4994bc5',) >Thread-18::INFO::2014-01-22 16:38:06,072::nfsSD::69::Storage.StorageDomain::(create) sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5 domainName=hosted_storage remotePath=10.34.63.199:/jbelka/hosted domClass=1 >Thread-18::DEBUG::2014-01-22 16:38:06,156::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-18::DEBUG::2014-01-22 16:38:06,163::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-18::WARNING::2014-01-22 16:38:06,163::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-18::DEBUG::2014-01-22 16:38:06,163::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-18::DEBUG::2014-01-22 16:38:06,164::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-18::DEBUG::2014-01-22 16:38:06,164::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Regular', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=544cc8c5919807ccac3ec08541e0233fea15ea88'] >Thread-18::DEBUG::2014-01-22 16:38:06,168::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-18::DEBUG::2014-01-22 16:38:06,169::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-18::DEBUG::2014-01-22 16:38:06,170::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-18::DEBUG::2014-01-22 16:38:06,178::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Regular', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=544cc8c5919807ccac3ec08541e0233fea15ea88'] >Thread-18::DEBUG::2014-01-22 16:38:06,182::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-18::DEBUG::2014-01-22 16:38:06,183::resourceManager::420::ResourceManager::(registerNamespace) Registering namespace '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS' >Thread-18::DEBUG::2014-01-22 16:38:06,183::resourceManager::420::ResourceManager::(registerNamespace) Registering namespace '3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS' >Thread-18::DEBUG::2014-01-22 16:38:06,183::clusterlock::137::initSANLock::(initSANLock) Initializing SANLock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-18::DEBUG::2014-01-22 16:38:06,225::sd::424::Storage.StorageDomain::(initSPMlease) lease initialized successfully >Thread-18::DEBUG::2014-01-22 16:38:06,225::hsm::2637::Storage.HSM::(createStorageDomain) knownSDs: {3900e952-713e-4ea9-b32c-097cd4994bc5: storage.nfsSD.findDomain} >Thread-18::INFO::2014-01-22 16:38:06,225::logUtils::47::dispatcher::(wrapper) Run and protect: createStorageDomain, Return response: None >Thread-18::DEBUG::2014-01-22 16:38:06,226::task::1168::TaskManager.Task::(prepare) Task=`0e4b0cc9-5b42-4f2e-a93c-b71b9d4c15a1`::finished: None >Thread-18::DEBUG::2014-01-22 16:38:06,226::task::579::TaskManager.Task::(_updateState) Task=`0e4b0cc9-5b42-4f2e-a93c-b71b9d4c15a1`::moving from state preparing -> state finished >Thread-18::DEBUG::2014-01-22 16:38:06,226::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-18::DEBUG::2014-01-22 16:38:06,226::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-18::DEBUG::2014-01-22 16:38:06,227::task::974::TaskManager.Task::(_decref) Task=`0e4b0cc9-5b42-4f2e-a93c-b71b9d4c15a1`::ref 0 aborting False >Thread-20::DEBUG::2014-01-22 16:38:06,301::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-20::DEBUG::2014-01-22 16:38:06,306::task::579::TaskManager.Task::(_updateState) Task=`df6312d3-c789-4e8f-995e-d2e817e1768f`::moving from state init -> state preparing >Thread-20::INFO::2014-01-22 16:38:06,307::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-20::INFO::2014-01-22 16:38:06,307::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-20::DEBUG::2014-01-22 16:38:06,307::task::1168::TaskManager.Task::(prepare) Task=`df6312d3-c789-4e8f-995e-d2e817e1768f`::finished: {} >Thread-20::DEBUG::2014-01-22 16:38:06,307::task::579::TaskManager.Task::(_updateState) Task=`df6312d3-c789-4e8f-995e-d2e817e1768f`::moving from state preparing -> state finished >Thread-20::DEBUG::2014-01-22 16:38:06,307::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-20::DEBUG::2014-01-22 16:38:06,308::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-20::DEBUG::2014-01-22 16:38:06,308::task::974::TaskManager.Task::(_decref) Task=`df6312d3-c789-4e8f-995e-d2e817e1768f`::ref 0 aborting False >Thread-21::DEBUG::2014-01-22 16:38:06,362::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-21::DEBUG::2014-01-22 16:38:06,362::task::579::TaskManager.Task::(_updateState) Task=`1c992562-dfd3-4bb1-b331-413659a861ad`::moving from state init -> state preparing >Thread-21::INFO::2014-01-22 16:38:06,363::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainStats(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', options=None) >Thread-21::DEBUG::2014-01-22 16:38:06,363::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`9e0a54c0-4ade-45d8-9b0b-1c39b56bc10e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2800' at 'getStorageDomainStats' >Thread-21::DEBUG::2014-01-22 16:38:06,363::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'shared' >Thread-21::DEBUG::2014-01-22 16:38:06,364::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'shared' (1 active user) >Thread-21::DEBUG::2014-01-22 16:38:06,364::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`9e0a54c0-4ade-45d8-9b0b-1c39b56bc10e`::Granted request >Thread-21::DEBUG::2014-01-22 16:38:06,364::task::811::TaskManager.Task::(resourceAcquired) Task=`1c992562-dfd3-4bb1-b331-413659a861ad`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (shared) >Thread-21::DEBUG::2014-01-22 16:38:06,365::task::974::TaskManager.Task::(_decref) Task=`1c992562-dfd3-4bb1-b331-413659a861ad`::ref 1 aborting False >Thread-21::INFO::2014-01-22 16:38:06,366::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainStats, Return response: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '75298668544', 'disktotal': '422943129600', 'mdafree': 0}} >Thread-21::DEBUG::2014-01-22 16:38:06,366::task::1168::TaskManager.Task::(prepare) Task=`1c992562-dfd3-4bb1-b331-413659a861ad`::finished: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '75298668544', 'disktotal': '422943129600', 'mdafree': 0}} >Thread-21::DEBUG::2014-01-22 16:38:06,366::task::579::TaskManager.Task::(_updateState) Task=`1c992562-dfd3-4bb1-b331-413659a861ad`::moving from state preparing -> state finished >Thread-21::DEBUG::2014-01-22 16:38:06,367::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>} >Thread-21::DEBUG::2014-01-22 16:38:06,367::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-21::DEBUG::2014-01-22 16:38:06,367::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >Thread-21::DEBUG::2014-01-22 16:38:06,367::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >Thread-21::DEBUG::2014-01-22 16:38:06,368::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >Thread-21::DEBUG::2014-01-22 16:38:06,368::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >Thread-21::DEBUG::2014-01-22 16:38:06,368::task::974::TaskManager.Task::(_decref) Task=`1c992562-dfd3-4bb1-b331-413659a861ad`::ref 0 aborting False >Thread-22::DEBUG::2014-01-22 16:38:06,422::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-22::DEBUG::2014-01-22 16:38:06,422::task::579::TaskManager.Task::(_updateState) Task=`73f41fb8-31b3-40ec-b0ac-7610326ccfb3`::moving from state init -> state preparing >Thread-22::INFO::2014-01-22 16:38:06,423::logUtils::44::dispatcher::(wrapper) Run and protect: createStoragePool(poolType=None, spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', poolName='hosted_datacenter', masterDom='3900e952-713e-4ea9-b32c-097cd4994bc5', domList=['3900e952-713e-4ea9-b32c-097cd4994bc5'], masterVersion=1, lockPolicy=None, lockRenewalIntervalSec=None, leaseTimeSec=None, ioOpTimeoutSec=None, leaseRetries=None, options=None) >Thread-22::INFO::2014-01-22 16:38:06,423::fileSD::299::Storage.StorageDomain::(validate) sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-22::DEBUG::2014-01-22 16:38:06,432::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Regular', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=544cc8c5919807ccac3ec08541e0233fea15ea88'] >Thread-22::DEBUG::2014-01-22 16:38:06,432::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`d8a77a81-b704-4f39-9394-465c25cd6a98`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '977' at 'createStoragePool' >Thread-22::DEBUG::2014-01-22 16:38:06,433::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-22::DEBUG::2014-01-22 16:38:06,433::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-22::DEBUG::2014-01-22 16:38:06,434::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`d8a77a81-b704-4f39-9394-465c25cd6a98`::Granted request >Thread-22::DEBUG::2014-01-22 16:38:06,434::task::811::TaskManager.Task::(resourceAcquired) Task=`73f41fb8-31b3-40ec-b0ac-7610326ccfb3`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-22::DEBUG::2014-01-22 16:38:06,435::task::974::TaskManager.Task::(_decref) Task=`73f41fb8-31b3-40ec-b0ac-7610326ccfb3`::ref 1 aborting False >Thread-22::DEBUG::2014-01-22 16:38:06,435::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`27b57ade-02a7-49a5-92be-a8719f222860`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '979' at 'createStoragePool' >Thread-22::DEBUG::2014-01-22 16:38:06,435::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'exclusive' >Thread-22::DEBUG::2014-01-22 16:38:06,436::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'exclusive' (1 active user) >Thread-22::DEBUG::2014-01-22 16:38:06,436::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`27b57ade-02a7-49a5-92be-a8719f222860`::Granted request >Thread-22::DEBUG::2014-01-22 16:38:06,436::task::811::TaskManager.Task::(resourceAcquired) Task=`73f41fb8-31b3-40ec-b0ac-7610326ccfb3`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (exclusive) >Thread-22::DEBUG::2014-01-22 16:38:06,436::task::974::TaskManager.Task::(_decref) Task=`73f41fb8-31b3-40ec-b0ac-7610326ccfb3`::ref 1 aborting False >Thread-22::INFO::2014-01-22 16:38:06,437::sp::586::Storage.StoragePool::(create) spUUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb poolName=hosted_datacenter master_sd=3900e952-713e-4ea9-b32c-097cd4994bc5 domList=['3900e952-713e-4ea9-b32c-097cd4994bc5'] masterVersion=1 {'LEASETIMESEC': 60, 'LOCKPOLICY': 'ON', 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5} >Thread-22::INFO::2014-01-22 16:38:06,437::fileSD::299::Storage.StorageDomain::(validate) sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-22::DEBUG::2014-01-22 16:38:06,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Regular', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=544cc8c5919807ccac3ec08541e0233fea15ea88'] >Thread-22::DEBUG::2014-01-22 16:38:06,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Regular', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=544cc8c5919807ccac3ec08541e0233fea15ea88'] >Thread-22::DEBUG::2014-01-22 16:38:06,449::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-22::DEBUG::2014-01-22 16:38:06,449::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-22::DEBUG::2014-01-22 16:38:06,450::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Regular', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8122e8f1205391147b95bb5c9dc2da7d82290d58'] >Thread-22::DEBUG::2014-01-22 16:38:06,455::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-22::INFO::2014-01-22 16:38:06,455::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 250) >Thread-22::DEBUG::2014-01-22 16:38:27,460::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 250) >Thread-22::INFO::2014-01-22 16:38:27,467::clusterlock::225::SANLock::(acquire) Acquiring cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 250) >Thread-22::DEBUG::2014-01-22 16:38:27,509::clusterlock::251::SANLock::(acquire) Cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 250) >Thread-22::INFO::2014-01-22 16:38:27,509::sp::754::Storage.StoragePool::(createMaster) setting master domain for spUUID e1263145-a6a9-4993-9d8a-cd92a3e5c1fb on sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-22::DEBUG::2014-01-22 16:38:27,509::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-22::DEBUG::2014-01-22 16:38:27,510::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-22::DEBUG::2014-01-22 16:38:27,522::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Regular', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8122e8f1205391147b95bb5c9dc2da7d82290d58'] >Thread-22::DEBUG::2014-01-22 16:38:27,523::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-22::DEBUG::2014-01-22 16:38:27,523::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-22::DEBUG::2014-01-22 16:38:27,523::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-22::DEBUG::2014-01-22 16:38:27,523::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-22::DEBUG::2014-01-22 16:38:27,523::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=309f8432a6387aa9776d65d504ed4a6d718ce02b'] >Thread-22::DEBUG::2014-01-22 16:38:27,528::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-22::DEBUG::2014-01-22 16:38:27,528::sp::1529::Storage.StoragePool::(getMasterDomain) Master domain 3900e952-713e-4ea9-b32c-097cd4994bc5 verified, version 1 >Thread-22::DEBUG::2014-01-22 16:38:27,528::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-22::DEBUG::2014-01-22 16:38:27,529::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-22::DEBUG::2014-01-22 16:38:27,537::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=309f8432a6387aa9776d65d504ed4a6d718ce02b'] >Thread-22::INFO::2014-01-22 16:38:27,537::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-23::DEBUG::2014-01-22 16:38:27,537::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-22::DEBUG::2014-01-22 16:38:27,538::sp::1562::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `e1263145-a6a9-4993-9d8a-cd92a3e5c1fb` started monitoring domain `3900e952-713e-4ea9-b32c-097cd4994bc5` >Thread-22::DEBUG::2014-01-22 16:38:27,538::misc::819::SamplingMethod::(__call__) Returning last result >Thread-22::WARNING::2014-01-22 16:38:27,539::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb already exists >Thread-22::DEBUG::2014-01-22 16:38:27,541::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2014-01-22 16:38:27,542::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-23::DEBUG::2014-01-22 16:38:27,555::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-23::DEBUG::2014-01-22 16:38:27,561::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n481 bytes (481 B) copied, 0.000310534 s, 1.5 MB/s\n'; <rc> = 0 >Thread-23::DEBUG::2014-01-22 16:38:27,563::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3900e952-713e-4ea9-b32c-097cd4994bc5 changed its status to Valid >Thread-22::DEBUG::2014-01-22 16:38:27,672::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >Thread-22::DEBUG::2014-01-22 16:38:27,674::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::INFO::2014-01-22 16:38:27,678::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-22::INFO::2014-01-22 16:38:27,679::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd >Thread-22::INFO::2014-01-22 16:38:27,679::clusterlock::255::SANLock::(release) Releasing cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-22::DEBUG::2014-01-22 16:38:27,682::clusterlock::266::SANLock::(release) Cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully released >Thread-22::INFO::2014-01-22 16:38:27,682::clusterlock::197::SANLock::(releaseHostId) Releasing host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 250) >Thread-22::DEBUG::2014-01-22 16:38:29,684::clusterlock::207::SANLock::(releaseHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 released successfully (id: 250) >Thread-22::INFO::2014-01-22 16:38:29,684::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-23::DEBUG::2014-01-22 16:38:29,685::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-23::INFO::2014-01-22 16:38:29,685::clusterlock::197::SANLock::(releaseHostId) Releasing host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 250) >Thread-23::DEBUG::2014-01-22 16:38:29,686::clusterlock::207::SANLock::(releaseHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 released successfully (id: 250) >Thread-22::INFO::2014-01-22 16:38:29,686::logUtils::47::dispatcher::(wrapper) Run and protect: createStoragePool, Return response: True >Thread-22::DEBUG::2014-01-22 16:38:29,687::task::1168::TaskManager.Task::(prepare) Task=`73f41fb8-31b3-40ec-b0ac-7610326ccfb3`::finished: True >Thread-22::DEBUG::2014-01-22 16:38:29,687::task::579::TaskManager.Task::(_updateState) Task=`73f41fb8-31b3-40ec-b0ac-7610326ccfb3`::moving from state preparing -> state finished >Thread-22::DEBUG::2014-01-22 16:38:29,687::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>, 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-22::DEBUG::2014-01-22 16:38:29,687::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-22::DEBUG::2014-01-22 16:38:29,688::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >Thread-22::DEBUG::2014-01-22 16:38:29,688::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >Thread-22::DEBUG::2014-01-22 16:38:29,688::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >Thread-22::DEBUG::2014-01-22 16:38:29,688::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >Thread-22::DEBUG::2014-01-22 16:38:29,689::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-22::DEBUG::2014-01-22 16:38:29,689::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-22::DEBUG::2014-01-22 16:38:29,689::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-22::DEBUG::2014-01-22 16:38:29,690::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-22::DEBUG::2014-01-22 16:38:29,690::task::974::TaskManager.Task::(_decref) Task=`73f41fb8-31b3-40ec-b0ac-7610326ccfb3`::ref 0 aborting False >Thread-25::DEBUG::2014-01-22 16:38:29,764::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-25::DEBUG::2014-01-22 16:38:29,765::task::579::TaskManager.Task::(_updateState) Task=`781b8d6c-3f5c-4ab1-8614-b95d26a3a3a9`::moving from state init -> state preparing >Thread-25::INFO::2014-01-22 16:38:29,765::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', hostID=1, scsiKey='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', msdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', masterVersion=1, options=None) >Thread-25::DEBUG::2014-01-22 16:38:29,766::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`dc9e2957-2e34-4d27-9ef6-caf770067393`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool' >Thread-25::DEBUG::2014-01-22 16:38:29,766::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-25::DEBUG::2014-01-22 16:38:29,766::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-25::DEBUG::2014-01-22 16:38:29,767::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`dc9e2957-2e34-4d27-9ef6-caf770067393`::Granted request >Thread-25::DEBUG::2014-01-22 16:38:29,768::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`68272ce8-61cd-488e-9ec2-283ea36252ef`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool' >Thread-25::DEBUG::2014-01-22 16:38:29,768::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-25::DEBUG::2014-01-22 16:38:29,768::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-25::DEBUG::2014-01-22 16:38:29,768::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`68272ce8-61cd-488e-9ec2-283ea36252ef`::Granted request >Thread-25::INFO::2014-01-22 16:38:29,769::sp::674::Storage.StoragePool::(connect) Connect host #1 to the storage pool e1263145-a6a9-4993-9d8a-cd92a3e5c1fb with master domain: 3900e952-713e-4ea9-b32c-097cd4994bc5 (ver = 1) >Thread-25::DEBUG::2014-01-22 16:38:29,769::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:29,769::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:29,770::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:29,770::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:29,770::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:29,770::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:29,770::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-25::DEBUG::2014-01-22 16:38:29,770::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-25::DEBUG::2014-01-22 16:38:29,770::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-25::DEBUG::2014-01-22 16:38:29,770::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-25::DEBUG::2014-01-22 16:38:29,771::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-25::DEBUG::2014-01-22 16:38:29,782::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-25::DEBUG::2014-01-22 16:38:29,783::misc::819::SamplingMethod::(__call__) Returning last result >Thread-25::DEBUG::2014-01-22 16:38:33,846::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-25::DEBUG::2014-01-22 16:38:33,977::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-25::DEBUG::2014-01-22 16:38:33,978::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:33,978::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:33,979::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:33,979::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:33,979::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:33,980::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:33,980::misc::819::SamplingMethod::(__call__) Returning last result >Thread-25::DEBUG::2014-01-22 16:38:33,986::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-25::DEBUG::2014-01-22 16:38:33,988::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-25::DEBUG::2014-01-22 16:38:33,996::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=309f8432a6387aa9776d65d504ed4a6d718ce02b'] >Thread-25::DEBUG::2014-01-22 16:38:33,998::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-25::INFO::2014-01-22 16:38:33,998::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS already registered >Thread-25::INFO::2014-01-22 16:38:33,998::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS already registered >Thread-25::DEBUG::2014-01-22 16:38:33,998::sp::1529::Storage.StoragePool::(getMasterDomain) Master domain 3900e952-713e-4ea9-b32c-097cd4994bc5 verified, version 1 >Thread-25::DEBUG::2014-01-22 16:38:33,999::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-25::DEBUG::2014-01-22 16:38:33,999::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-25::DEBUG::2014-01-22 16:38:34,006::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=309f8432a6387aa9776d65d504ed4a6d718ce02b'] >Thread-25::INFO::2014-01-22 16:38:34,006::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-25::DEBUG::2014-01-22 16:38:34,007::sp::1562::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `e1263145-a6a9-4993-9d8a-cd92a3e5c1fb` started monitoring domain `3900e952-713e-4ea9-b32c-097cd4994bc5` >Thread-27::DEBUG::2014-01-22 16:38:34,007::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-25::DEBUG::2014-01-22 16:38:34,008::misc::819::SamplingMethod::(__call__) Returning last result >Thread-25::WARNING::2014-01-22 16:38:34,009::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb already exists >Thread-25::DEBUG::2014-01-22 16:38:34,013::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-25::DEBUG::2014-01-22 16:38:34,022::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-27::DEBUG::2014-01-22 16:38:34,033::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-27::DEBUG::2014-01-22 16:38:34,037::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n481 bytes (481 B) copied, 0.000253023 s, 1.9 MB/s\n'; <rc> = 0 >Thread-27::DEBUG::2014-01-22 16:38:34,039::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3900e952-713e-4ea9-b32c-097cd4994bc5 changed its status to Valid >Thread-28::DEBUG::2014-01-22 16:38:34,040::misc::931::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-27::INFO::2014-01-22 16:38:34,040::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-28::DEBUG::2014-01-22 16:38:34,040::misc::941::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-27::DEBUG::2014-01-22 16:38:34,041::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >Thread-28::DEBUG::2014-01-22 16:38:34,043::misc::951::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-29::INFO::2014-01-22 16:38:34,048::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-25::DEBUG::2014-01-22 16:38:34,137::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >Thread-25::DEBUG::2014-01-22 16:38:34,138::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-25::INFO::2014-01-22 16:38:34,140::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-25::INFO::2014-01-22 16:38:34,141::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd >Thread-25::DEBUG::2014-01-22 16:38:34,141::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-25::DEBUG::2014-01-22 16:38:34,142::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-25::DEBUG::2014-01-22 16:38:34,142::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-25::DEBUG::2014-01-22 16:38:34,142::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-25::DEBUG::2014-01-22 16:38:34,143::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-25::DEBUG::2014-01-22 16:38:34,143::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-25::DEBUG::2014-01-22 16:38:34,143::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-25::DEBUG::2014-01-22 16:38:34,143::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-25::INFO::2014-01-22 16:38:34,144::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-25::DEBUG::2014-01-22 16:38:34,144::task::1168::TaskManager.Task::(prepare) Task=`781b8d6c-3f5c-4ab1-8614-b95d26a3a3a9`::finished: True >Thread-25::DEBUG::2014-01-22 16:38:34,144::task::579::TaskManager.Task::(_updateState) Task=`781b8d6c-3f5c-4ab1-8614-b95d26a3a3a9`::moving from state preparing -> state finished >Thread-25::DEBUG::2014-01-22 16:38:34,144::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25::DEBUG::2014-01-22 16:38:34,145::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25::DEBUG::2014-01-22 16:38:34,145::task::974::TaskManager.Task::(_decref) Task=`781b8d6c-3f5c-4ab1-8614-b95d26a3a3a9`::ref 0 aborting False >Thread-31::DEBUG::2014-01-22 16:38:34,202::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-31::DEBUG::2014-01-22 16:38:34,203::task::579::TaskManager.Task::(_updateState) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::moving from state init -> state preparing >Thread-31::INFO::2014-01-22 16:38:34,203::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', prevID=-1, prevLVER=-1, recoveryMode=None, scsiFencing='false', maxHostID=250, domVersion=3, options=None) >Thread-31::DEBUG::2014-01-22 16:38:34,204::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`dfda044c-dc87-444a-9c48-8e939b2b7e58`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '587' at 'spmStart' >Thread-31::DEBUG::2014-01-22 16:38:34,204::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-31::DEBUG::2014-01-22 16:38:34,204::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-31::DEBUG::2014-01-22 16:38:34,204::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`dfda044c-dc87-444a-9c48-8e939b2b7e58`::Granted request >Thread-31::DEBUG::2014-01-22 16:38:34,205::task::811::TaskManager.Task::(resourceAcquired) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-31::DEBUG::2014-01-22 16:38:34,205::task::974::TaskManager.Task::(_decref) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::ref 1 aborting False >Thread-31::INFO::2014-01-22 16:38:34,205::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-31::DEBUG::2014-01-22 16:38:34,206::task::1163::TaskManager.Task::(prepare) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::Prepare: 1 jobs exist, move to acquiring >Thread-31::DEBUG::2014-01-22 16:38:34,206::task::579::TaskManager.Task::(_updateState) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::moving from state preparing -> state acquiring >Thread-31::DEBUG::2014-01-22 16:38:34,206::task::579::TaskManager.Task::(_updateState) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::moving from state acquiring -> state queued >Thread-31::DEBUG::2014-01-22 16:38:34,206::taskManager::50::TaskManager::(_queueTask) queuing task: bb12b4bc-efe3-4668-88c8-2fef263aa553 >Thread-31::DEBUG::2014-01-22 16:38:34,207::taskManager::56::TaskManager::(_queueTask) task queued: bb12b4bc-efe3-4668-88c8-2fef263aa553 >Thread-1::DEBUG::2014-01-22 16:38:34,207::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-31::DEBUG::2014-01-22 16:38:34,208::task::1165::TaskManager.Task::(prepare) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::returning >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,208::threadPool::205::Misc.ThreadPool.WorkerThread::(run) Task: bb12b4bc-efe3-4668-88c8-2fef263aa553 running: <bound method Task.commit of <storage.task.Task instance at 0x146ca70>> with: None >Thread-31::DEBUG::2014-01-22 16:38:34,209::task::974::TaskManager.Task::(_decref) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::ref 0 aborting False >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,209::task::1176::TaskManager.Task::(commit) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::committing task: bb12b4bc-efe3-4668-88c8-2fef263aa553 >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,210::task::579::TaskManager.Task::(_updateState) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::moving from state queued -> state running >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,211::task::889::TaskManager.Task::(_runJobs) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x147b290>> (args: (-1, -1, 'false', 250, 3) kwargs: {}) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,212::task::315::TaskManager.Task::(run) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x147b290>> (args: (-1, -1, 'false', 250, 3) kwargs: {}) callback None >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,212::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,213::misc::811::SamplingMethod::(__call__) Got in to sampling method >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,217::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=309f8432a6387aa9776d65d504ed4a6d718ce02b'] >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,218::misc::819::SamplingMethod::(__call__) Returning last result >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:34,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=309f8432a6387aa9776d65d504ed4a6d718ce02b'] >bb12b4bc-efe3-4668-88c8-2fef263aa553::INFO::2014-01-22 16:38:34,224::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-32::DEBUG::2014-01-22 16:38:34,259::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-32::DEBUG::2014-01-22 16:38:34,260::task::579::TaskManager.Task::(_updateState) Task=`531a1877-de87-4c0f-baf7-a467ab7a347b`::moving from state init -> state preparing >Thread-32::INFO::2014-01-22 16:38:34,260::logUtils::44::dispatcher::(wrapper) Run and protect: activateStorageDomain(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-32::DEBUG::2014-01-22 16:38:34,260::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`a452fffc-eda8-451c-a500-3bc580c6c247`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1239' at 'activateStorageDomain' >Thread-32::DEBUG::2014-01-22 16:38:34,261::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-32::DEBUG::2014-01-22 16:38:34,261::resourceManager::576::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is currently locked, Entering queue (1 in queue) >Thread-27::DEBUG::2014-01-22 16:38:44,056::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-27::DEBUG::2014-01-22 16:38:44,063::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n481 bytes (481 B) copied, 0.000379546 s, 1.3 MB/s\n'; <rc> = 0 >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,227::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >bb12b4bc-efe3-4668-88c8-2fef263aa553::INFO::2014-01-22 16:38:55,228::clusterlock::225::SANLock::(acquire) Acquiring cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,262::clusterlock::251::SANLock::(acquire) Cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,262::sp::272::Storage.StoragePool::(startSpm) spm lock acquired successfully >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,270::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=309f8432a6387aa9776d65d504ed4a6d718ce02b'] >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,270::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,270::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,271::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=162a9510b0b9de95826abed1e4f61da33fca7ccb'] >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,276::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,276::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`9eab9221-1eb1-4797-9ea1-603eff357181`::Request was made in '/usr/share/vdsm/storage/sp.py' line '432' at '_upgradePool' >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,277::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,277::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,277::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`9eab9221-1eb1-4797-9ea1-603eff357181`::Granted request >bb12b4bc-efe3-4668-88c8-2fef263aa553::INFO::2014-01-22 16:38:55,278::sp::438::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `3900e952-713e-4ea9-b32c-097cd4994bc5` >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,278::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`4ee7754a-ce7f-45de-9028-3ff926e1a9b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '440' at '_upgradePool' >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,278::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'exclusive' >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,278::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'exclusive' (1 active user) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,279::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`4ee7754a-ce7f-45de-9028-3ff926e1a9b4`::Granted request >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,283::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,284::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,292::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=162a9510b0b9de95826abed1e4f61da33fca7ccb'] >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,294::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >bb12b4bc-efe3-4668-88c8-2fef263aa553::INFO::2014-01-22 16:38:55,294::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS already registered >bb12b4bc-efe3-4668-88c8-2fef263aa553::INFO::2014-01-22 16:38:55,294::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS already registered >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,295::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,295::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,295::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,295::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,295::sp::443::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,296::sp::450::Storage.StoragePool::(_upgradePool) Registering with state change event >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,296::sp::453::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,296::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,296::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,296::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,297::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,300::sd::620::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/master/vms >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,301::sd::623::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/master/tasks >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,304::sp::333::Storage.StoragePool::(startSpm) ended. >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,304::task::579::TaskManager.Task::(_updateState) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::moving from state running -> state finished >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,304::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,304::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,305::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,305::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,305::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,305::resourceManager::653::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' has 1 requests in queue. Handling top request. >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,305::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`a452fffc-eda8-451c-a500-3bc580c6c247`::Granted request >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,306::resourceManager::685::ResourceManager::(releaseResource) Request 'Request for Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb - exclusive: granted' was granted >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,306::task::897::TaskManager.Task::(_runJobs) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::Task.run: exit - success: result >Thread-32::DEBUG::2014-01-22 16:38:55,307::task::811::TaskManager.Task::(resourceAcquired) Task=`531a1877-de87-4c0f-baf7-a467ab7a347b`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,307::task::974::TaskManager.Task::(_decref) Task=`bb12b4bc-efe3-4668-88c8-2fef263aa553`::ref 0 aborting False >Thread-32::DEBUG::2014-01-22 16:38:55,307::task::974::TaskManager.Task::(_decref) Task=`531a1877-de87-4c0f-baf7-a467ab7a347b`::ref 1 aborting False >bb12b4bc-efe3-4668-88c8-2fef263aa553::DEBUG::2014-01-22 16:38:55,308::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-32::DEBUG::2014-01-22 16:38:55,309::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`79e1d076-0c25-4303-a36e-4df73bbc5c53`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1240' at 'activateStorageDomain' >Thread-32::DEBUG::2014-01-22 16:38:55,309::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'exclusive' >Thread-32::DEBUG::2014-01-22 16:38:55,309::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'exclusive' (1 active user) >Thread-32::DEBUG::2014-01-22 16:38:55,309::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`79e1d076-0c25-4303-a36e-4df73bbc5c53`::Granted request >Thread-32::DEBUG::2014-01-22 16:38:55,310::task::811::TaskManager.Task::(resourceAcquired) Task=`531a1877-de87-4c0f-baf7-a467ab7a347b`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (exclusive) >Thread-32::DEBUG::2014-01-22 16:38:55,310::task::974::TaskManager.Task::(_decref) Task=`531a1877-de87-4c0f-baf7-a467ab7a347b`::ref 1 aborting False >Thread-32::INFO::2014-01-22 16:38:55,310::sp::1106::Storage.StoragePool::(activateSD) sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5 spUUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb >Thread-32::INFO::2014-01-22 16:38:55,311::logUtils::47::dispatcher::(wrapper) Run and protect: activateStorageDomain, Return response: None >Thread-32::DEBUG::2014-01-22 16:38:55,311::task::1168::TaskManager.Task::(prepare) Task=`531a1877-de87-4c0f-baf7-a467ab7a347b`::finished: None >Thread-32::DEBUG::2014-01-22 16:38:55,311::task::579::TaskManager.Task::(_updateState) Task=`531a1877-de87-4c0f-baf7-a467ab7a347b`::moving from state preparing -> state finished >Thread-32::DEBUG::2014-01-22 16:38:55,311::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>, 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-32::DEBUG::2014-01-22 16:38:55,311::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32::DEBUG::2014-01-22 16:38:55,312::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >Thread-32::DEBUG::2014-01-22 16:38:55,312::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >Thread-32::DEBUG::2014-01-22 16:38:55,312::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >Thread-32::DEBUG::2014-01-22 16:38:55,312::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >Thread-32::DEBUG::2014-01-22 16:38:55,312::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-32::DEBUG::2014-01-22 16:38:55,313::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-32::DEBUG::2014-01-22 16:38:55,313::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-32::DEBUG::2014-01-22 16:38:55,313::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-32::DEBUG::2014-01-22 16:38:55,313::task::974::TaskManager.Task::(_decref) Task=`531a1877-de87-4c0f-baf7-a467ab7a347b`::ref 0 aborting False >Thread-34::DEBUG::2014-01-22 16:38:55,363::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-34::DEBUG::2014-01-22 16:38:55,363::task::579::TaskManager.Task::(_updateState) Task=`3e65d3b6-77b4-4601-83e1-448abc475b54`::moving from state init -> state preparing >Thread-34::INFO::2014-01-22 16:38:55,364::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-34::DEBUG::2014-01-22 16:38:55,364::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-34::DEBUG::2014-01-22 16:38:55,364::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-34::INFO::2014-01-22 16:38:55,364::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-34::DEBUG::2014-01-22 16:38:55,364::task::1168::TaskManager.Task::(prepare) Task=`3e65d3b6-77b4-4601-83e1-448abc475b54`::finished: {'allTasksStatus': {}} >Thread-34::DEBUG::2014-01-22 16:38:55,365::task::579::TaskManager.Task::(_updateState) Task=`3e65d3b6-77b4-4601-83e1-448abc475b54`::moving from state preparing -> state finished >Thread-34::DEBUG::2014-01-22 16:38:55,365::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34::DEBUG::2014-01-22 16:38:55,365::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34::DEBUG::2014-01-22 16:38:55,365::task::974::TaskManager.Task::(_decref) Task=`3e65d3b6-77b4-4601-83e1-448abc475b54`::ref 0 aborting False >Thread-35::DEBUG::2014-01-22 16:38:55,416::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-35::DEBUG::2014-01-22 16:38:55,416::task::579::TaskManager.Task::(_updateState) Task=`b8c76095-2b64-4bc1-907d-0a240aca4250`::moving from state init -> state preparing >Thread-35::INFO::2014-01-22 16:38:55,417::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-35::INFO::2014-01-22 16:38:55,417::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-35::DEBUG::2014-01-22 16:38:55,417::task::1168::TaskManager.Task::(prepare) Task=`b8c76095-2b64-4bc1-907d-0a240aca4250`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-35::DEBUG::2014-01-22 16:38:55,417::task::579::TaskManager.Task::(_updateState) Task=`b8c76095-2b64-4bc1-907d-0a240aca4250`::moving from state preparing -> state finished >Thread-35::DEBUG::2014-01-22 16:38:55,418::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35::DEBUG::2014-01-22 16:38:55,418::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35::DEBUG::2014-01-22 16:38:55,418::task::974::TaskManager.Task::(_decref) Task=`b8c76095-2b64-4bc1-907d-0a240aca4250`::ref 0 aborting False >Thread-36::DEBUG::2014-01-22 16:38:55,469::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-36::DEBUG::2014-01-22 16:38:55,469::task::579::TaskManager.Task::(_updateState) Task=`01a7c93a-998b-4ea0-a9a5-c3a65da5e80a`::moving from state init -> state preparing >Thread-36::INFO::2014-01-22 16:38:55,470::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-36::DEBUG::2014-01-22 16:38:55,470::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`9dd2fa19-6b0d-402b-9fce-3db15c49a524`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2562' at 'getStoragePoolInfo' >Thread-36::DEBUG::2014-01-22 16:38:55,470::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'shared' >Thread-36::DEBUG::2014-01-22 16:38:55,471::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'shared' (1 active user) >Thread-36::DEBUG::2014-01-22 16:38:55,471::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`9dd2fa19-6b0d-402b-9fce-3db15c49a524`::Granted request >Thread-36::DEBUG::2014-01-22 16:38:55,471::task::811::TaskManager.Task::(resourceAcquired) Task=`01a7c93a-998b-4ea0-a9a5-c3a65da5e80a`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (shared) >Thread-36::DEBUG::2014-01-22 16:38:55,471::task::974::TaskManager.Task::(_decref) Task=`01a7c93a-998b-4ea0-a9a5-c3a65da5e80a`::ref 1 aborting False >Thread-36::INFO::2014-01-22 16:38:55,474::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'name': 'hosted_datacenter', 'version': '3', 'domains': '3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 0}, 'dominfo': {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'status': 'Active', 'diskfree': '75298668544', 'isoprefix': '', 'alerts': [], 'disktotal': '422943129600', 'version': 3}}} >Thread-36::DEBUG::2014-01-22 16:38:55,475::task::1168::TaskManager.Task::(prepare) Task=`01a7c93a-998b-4ea0-a9a5-c3a65da5e80a`::finished: {'info': {'spm_id': 1, 'master_uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'name': 'hosted_datacenter', 'version': '3', 'domains': '3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 0}, 'dominfo': {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'status': 'Active', 'diskfree': '75298668544', 'isoprefix': '', 'alerts': [], 'disktotal': '422943129600', 'version': 3}}} >Thread-36::DEBUG::2014-01-22 16:38:55,475::task::579::TaskManager.Task::(_updateState) Task=`01a7c93a-998b-4ea0-a9a5-c3a65da5e80a`::moving from state preparing -> state finished >Thread-36::DEBUG::2014-01-22 16:38:55,475::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-36::DEBUG::2014-01-22 16:38:55,475::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-36::DEBUG::2014-01-22 16:38:55,476::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-36::DEBUG::2014-01-22 16:38:55,476::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-36::DEBUG::2014-01-22 16:38:55,476::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-36::DEBUG::2014-01-22 16:38:55,476::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-36::DEBUG::2014-01-22 16:38:55,477::task::974::TaskManager.Task::(_decref) Task=`01a7c93a-998b-4ea0-a9a5-c3a65da5e80a`::ref 0 aborting False >Thread-37::DEBUG::2014-01-22 16:38:55,529::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-37::DEBUG::2014-01-22 16:38:55,529::task::579::TaskManager.Task::(_updateState) Task=`00a63262-d2cd-424d-a032-d6004b56259a`::moving from state init -> state preparing >Thread-37::INFO::2014-01-22 16:38:55,530::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-37::INFO::2014-01-22 16:38:55,530::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000379546', 'lastCheck': '0.3', 'valid': True}} >Thread-37::DEBUG::2014-01-22 16:38:55,530::task::1168::TaskManager.Task::(prepare) Task=`00a63262-d2cd-424d-a032-d6004b56259a`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000379546', 'lastCheck': '0.3', 'valid': True}} >Thread-37::DEBUG::2014-01-22 16:38:55,530::task::579::TaskManager.Task::(_updateState) Task=`00a63262-d2cd-424d-a032-d6004b56259a`::moving from state preparing -> state finished >Thread-37::DEBUG::2014-01-22 16:38:55,530::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-37::DEBUG::2014-01-22 16:38:55,531::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-37::DEBUG::2014-01-22 16:38:55,531::task::974::TaskManager.Task::(_decref) Task=`00a63262-d2cd-424d-a032-d6004b56259a`::ref 0 aborting False >Thread-38::DEBUG::2014-01-22 16:38:55,818::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-38::DEBUG::2014-01-22 16:38:55,825::task::579::TaskManager.Task::(_updateState) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::moving from state init -> state preparing >Thread-38::INFO::2014-01-22 16:38:55,825::logUtils::44::dispatcher::(wrapper) Run and protect: createVolume(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', imgUUID='6f59738c-caf5-4be8-98f4-27eacf7775a3', size=52428800, volFormat=5, preallocate=2, diskType=2, volUUID='150ad1c0-3297-47c5-b79d-2fe98d16e759', desc='Hosted Engine Image', srcImgUUID='00000000-0000-0000-0000-000000000000', srcVolUUID='00000000-0000-0000-0000-000000000000') >Thread-38::DEBUG::2014-01-22 16:38:55,825::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`93e24d19-cb23-4184-96c2-8ccc342ed26d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1484' at 'createVolume' >Thread-38::DEBUG::2014-01-22 16:38:55,826::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'shared' >Thread-38::DEBUG::2014-01-22 16:38:55,826::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'shared' (1 active user) >Thread-38::DEBUG::2014-01-22 16:38:55,826::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`93e24d19-cb23-4184-96c2-8ccc342ed26d`::Granted request >Thread-38::DEBUG::2014-01-22 16:38:55,827::task::811::TaskManager.Task::(resourceAcquired) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (shared) >Thread-38::DEBUG::2014-01-22 16:38:55,827::task::974::TaskManager.Task::(_decref) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::ref 1 aborting False >Thread-38::DEBUG::2014-01-22 16:38:55,832::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >Thread-38::DEBUG::2014-01-22 16:38:55,844::taskManager::68::TaskManager::(scheduleJob) scheduled job createVolume for task d200332c-64ae-4ece-8e9c-4a89d47ba317 >Thread-38::INFO::2014-01-22 16:38:55,844::logUtils::47::dispatcher::(wrapper) Run and protect: createVolume, Return response: None >Thread-38::DEBUG::2014-01-22 16:38:55,844::task::1163::TaskManager.Task::(prepare) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::Prepare: 1 jobs exist, move to acquiring >Thread-38::DEBUG::2014-01-22 16:38:55,845::task::579::TaskManager.Task::(_updateState) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::moving from state preparing -> state acquiring >Thread-38::DEBUG::2014-01-22 16:38:55,845::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >Thread-38::DEBUG::2014-01-22 16:38:55,860::task::579::TaskManager.Task::(_updateState) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::moving from state acquiring -> state queued >Thread-38::DEBUG::2014-01-22 16:38:55,860::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >Thread-38::DEBUG::2014-01-22 16:38:55,876::taskManager::50::TaskManager::(_queueTask) queuing task: d200332c-64ae-4ece-8e9c-4a89d47ba317 >Thread-38::DEBUG::2014-01-22 16:38:55,876::taskManager::56::TaskManager::(_queueTask) task queued: d200332c-64ae-4ece-8e9c-4a89d47ba317 >Thread-38::DEBUG::2014-01-22 16:38:55,877::task::1165::TaskManager.Task::(prepare) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::returning >Thread-2::DEBUG::2014-01-22 16:38:55,877::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-38::DEBUG::2014-01-22 16:38:55,877::task::974::TaskManager.Task::(_decref) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::ref 0 aborting False >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,878::threadPool::205::Misc.ThreadPool.WorkerThread::(run) Task: d200332c-64ae-4ece-8e9c-4a89d47ba317 running: <bound method Task.commit of <storage.task.Task instance at 0x14700e0>> with: None >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,878::task::1176::TaskManager.Task::(commit) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::committing task: d200332c-64ae-4ece-8e9c-4a89d47ba317 >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,879::task::579::TaskManager.Task::(_updateState) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::moving from state queued -> state running >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,880::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,895::task::889::TaskManager.Task::(_runJobs) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::Task.run: running job 0: createVolume: <bound method StoragePool.createVolume of <storage.sp.StoragePool object at 0x147b290>> (args: ('3900e952-713e-4ea9-b32c-097cd4994bc5', '6f59738c-caf5-4be8-98f4-27eacf7775a3', 52428800, 5, 2, 2, '150ad1c0-3297-47c5-b79d-2fe98d16e759', 'Hosted Engine Image', '00000000-0000-0000-0000-000000000000', '00000000-0000-0000-0000-000000000000') kwargs: {}) >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,895::task::315::TaskManager.Task::(run) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::Job.run: running createVolume: <bound method StoragePool.createVolume of <storage.sp.StoragePool object at 0x147b290>> (args: ('3900e952-713e-4ea9-b32c-097cd4994bc5', '6f59738c-caf5-4be8-98f4-27eacf7775a3', 52428800, 5, 2, 2, '150ad1c0-3297-47c5-b79d-2fe98d16e759', 'Hosted Engine Image', '00000000-0000-0000-0000-000000000000', '00000000-0000-0000-0000-000000000000') kwargs: {}) callback None >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,895::resourceManager::197::ResourceManager.Request::(__init__) ResName=`3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.6f59738c-caf5-4be8-98f4-27eacf7775a3`ReqID=`2f2b25a6-5c33-4fcd-bf7d-b844e0471d8d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '1895' at 'createVolume' >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,895::resourceManager::541::ResourceManager::(registerResource) Trying to register resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.6f59738c-caf5-4be8-98f4-27eacf7775a3' for lock type 'exclusive' >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,897::resourceFactories::125::Storage.ResourcesFactories::(__getResourceCandidatesList) Image 6f59738c-caf5-4be8-98f4-27eacf7775a3 does not exist in domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,897::resourceManager::600::ResourceManager::(registerResource) Resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.6f59738c-caf5-4be8-98f4-27eacf7775a3' is free. Now locking as 'exclusive' (1 active user) >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,897::resourceManager::237::ResourceManager.Request::(grant) ResName=`3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.6f59738c-caf5-4be8-98f4-27eacf7775a3`ReqID=`2f2b25a6-5c33-4fcd-bf7d-b844e0471d8d`::Granted request >d200332c-64ae-4ece-8e9c-4a89d47ba317::INFO::2014-01-22 16:38:55,897::image::123::Storage.Image::(create) Create placeholder /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3 for image's volumes >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,898::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >d200332c-64ae-4ece-8e9c-4a89d47ba317::INFO::2014-01-22 16:38:55,915::volume::446::Storage.Volume::(create) Creating volume 150ad1c0-3297-47c5-b79d-2fe98d16e759 >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,916::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >Thread-39::DEBUG::2014-01-22 16:38:55,932::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-39::DEBUG::2014-01-22 16:38:55,932::task::579::TaskManager.Task::(_updateState) Task=`4fcaa96d-5e3c-4aa3-9f38-84e8b413454a`::moving from state init -> state preparing >Thread-39::INFO::2014-01-22 16:38:55,932::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-39::DEBUG::2014-01-22 16:38:55,933::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-39::DEBUG::2014-01-22 16:38:55,933::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: d200332c-64ae-4ece-8e9c-4a89d47ba317 >Thread-39::DEBUG::2014-01-22 16:38:55,933::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd200332c-64ae-4ece-8e9c-4a89d47ba317'} >Thread-39::DEBUG::2014-01-22 16:38:55,933::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {'d200332c-64ae-4ece-8e9c-4a89d47ba317': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd200332c-64ae-4ece-8e9c-4a89d47ba317'}} >Thread-39::INFO::2014-01-22 16:38:55,933::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {'d200332c-64ae-4ece-8e9c-4a89d47ba317': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd200332c-64ae-4ece-8e9c-4a89d47ba317'}}} >Thread-39::DEBUG::2014-01-22 16:38:55,934::task::1168::TaskManager.Task::(prepare) Task=`4fcaa96d-5e3c-4aa3-9f38-84e8b413454a`::finished: {'allTasksStatus': {'d200332c-64ae-4ece-8e9c-4a89d47ba317': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd200332c-64ae-4ece-8e9c-4a89d47ba317'}}} >Thread-39::DEBUG::2014-01-22 16:38:55,934::task::579::TaskManager.Task::(_updateState) Task=`4fcaa96d-5e3c-4aa3-9f38-84e8b413454a`::moving from state preparing -> state finished >Thread-39::DEBUG::2014-01-22 16:38:55,934::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-39::DEBUG::2014-01-22 16:38:55,934::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-39::DEBUG::2014-01-22 16:38:55,934::task::974::TaskManager.Task::(_decref) Task=`4fcaa96d-5e3c-4aa3-9f38-84e8b413454a`::ref 0 aborting False >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,937::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >d200332c-64ae-4ece-8e9c-4a89d47ba317::INFO::2014-01-22 16:38:55,968::fileVolume::151::Storage.Volume::(_create) Request to create RAW volume /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759 with size = 52428800 sectors >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:55,971::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,002::fileVolume::382::Storage.Volume::(newVolumeLease) Initializing volume lease volUUID=150ad1c0-3297-47c5-b79d-2fe98d16e759 sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5, metaId=('/rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759',) >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,021::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,045::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.6f59738c-caf5-4be8-98f4-27eacf7775a3' >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,045::resourceManager::634::ResourceManager::(releaseResource) Released resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.6f59738c-caf5-4be8-98f4-27eacf7775a3' (0 active users) >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,046::resourceManager::640::ResourceManager::(releaseResource) Resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.6f59738c-caf5-4be8-98f4-27eacf7775a3' is free, finding out if anyone is waiting for it. >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,046::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.6f59738c-caf5-4be8-98f4-27eacf7775a3', Clearing records. >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,046::task::579::TaskManager.Task::(_updateState) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::moving from state running -> state finished >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,047::task::736::TaskManager.Task::(_save) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/d200332c-64ae-4ece-8e9c-4a89d47ba317.temp >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,071::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>} >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,071::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,072::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,072::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,072::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,072::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,073::task::897::TaskManager.Task::(_runJobs) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::Task.run: exit - success: result {'uuid': '150ad1c0-3297-47c5-b79d-2fe98d16e759'} >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,073::task::974::TaskManager.Task::(_decref) Task=`d200332c-64ae-4ece-8e9c-4a89d47ba317`::ref 0 aborting False >d200332c-64ae-4ece-8e9c-4a89d47ba317::DEBUG::2014-01-22 16:38:56,073::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-40::DEBUG::2014-01-22 16:38:56,987::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-40::DEBUG::2014-01-22 16:38:56,987::task::579::TaskManager.Task::(_updateState) Task=`bb7be910-0ec1-45b0-b669-3d0d86f40ff0`::moving from state init -> state preparing >Thread-40::INFO::2014-01-22 16:38:56,988::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-40::DEBUG::2014-01-22 16:38:56,988::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-40::DEBUG::2014-01-22 16:38:56,988::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: d200332c-64ae-4ece-8e9c-4a89d47ba317 >Thread-40::DEBUG::2014-01-22 16:38:56,988::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'd200332c-64ae-4ece-8e9c-4a89d47ba317'} >Thread-40::DEBUG::2014-01-22 16:38:56,988::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {'d200332c-64ae-4ece-8e9c-4a89d47ba317': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'd200332c-64ae-4ece-8e9c-4a89d47ba317'}} >Thread-40::INFO::2014-01-22 16:38:56,988::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {'d200332c-64ae-4ece-8e9c-4a89d47ba317': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'd200332c-64ae-4ece-8e9c-4a89d47ba317'}}} >Thread-40::DEBUG::2014-01-22 16:38:56,988::task::1168::TaskManager.Task::(prepare) Task=`bb7be910-0ec1-45b0-b669-3d0d86f40ff0`::finished: {'allTasksStatus': {'d200332c-64ae-4ece-8e9c-4a89d47ba317': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'd200332c-64ae-4ece-8e9c-4a89d47ba317'}}} >Thread-40::DEBUG::2014-01-22 16:38:56,988::task::579::TaskManager.Task::(_updateState) Task=`bb7be910-0ec1-45b0-b669-3d0d86f40ff0`::moving from state preparing -> state finished >Thread-40::DEBUG::2014-01-22 16:38:56,988::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-40::DEBUG::2014-01-22 16:38:56,988::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2014-01-22 16:38:56,989::task::974::TaskManager.Task::(_decref) Task=`bb7be910-0ec1-45b0-b669-3d0d86f40ff0`::ref 0 aborting False >Thread-41::DEBUG::2014-01-22 16:38:57,040::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-41::DEBUG::2014-01-22 16:38:57,040::task::579::TaskManager.Task::(_updateState) Task=`f985186f-5f68-427b-99dc-a0be32bf1ec6`::moving from state init -> state preparing >Thread-41::INFO::2014-01-22 16:38:57,041::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='d200332c-64ae-4ece-8e9c-4a89d47ba317', spUUID=None, options=None) >Thread-41::DEBUG::2014-01-22 16:38:57,041::taskManager::161::TaskManager::(clearTask) Entry. taskID: d200332c-64ae-4ece-8e9c-4a89d47ba317 >Thread-41::DEBUG::2014-01-22 16:38:57,049::taskManager::166::TaskManager::(clearTask) Return. >Thread-41::INFO::2014-01-22 16:38:57,049::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-41::DEBUG::2014-01-22 16:38:57,050::task::1168::TaskManager.Task::(prepare) Task=`f985186f-5f68-427b-99dc-a0be32bf1ec6`::finished: None >Thread-41::DEBUG::2014-01-22 16:38:57,050::task::579::TaskManager.Task::(_updateState) Task=`f985186f-5f68-427b-99dc-a0be32bf1ec6`::moving from state preparing -> state finished >Thread-41::DEBUG::2014-01-22 16:38:57,050::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-41::DEBUG::2014-01-22 16:38:57,050::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-41::DEBUG::2014-01-22 16:38:57,050::task::974::TaskManager.Task::(_decref) Task=`f985186f-5f68-427b-99dc-a0be32bf1ec6`::ref 0 aborting False >Thread-42::DEBUG::2014-01-22 16:38:57,103::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-42::DEBUG::2014-01-22 16:38:57,103::task::579::TaskManager.Task::(_updateState) Task=`3918c6d2-2245-4d31-810b-2a58af2513e9`::moving from state init -> state preparing >Thread-42::INFO::2014-01-22 16:38:57,104::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-42::DEBUG::2014-01-22 16:38:57,104::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-42::DEBUG::2014-01-22 16:38:57,104::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-42::INFO::2014-01-22 16:38:57,104::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-42::DEBUG::2014-01-22 16:38:57,104::task::1168::TaskManager.Task::(prepare) Task=`3918c6d2-2245-4d31-810b-2a58af2513e9`::finished: {'allTasksStatus': {}} >Thread-42::DEBUG::2014-01-22 16:38:57,104::task::579::TaskManager.Task::(_updateState) Task=`3918c6d2-2245-4d31-810b-2a58af2513e9`::moving from state preparing -> state finished >Thread-42::DEBUG::2014-01-22 16:38:57,105::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-42::DEBUG::2014-01-22 16:38:57,105::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-42::DEBUG::2014-01-22 16:38:57,105::task::974::TaskManager.Task::(_decref) Task=`3918c6d2-2245-4d31-810b-2a58af2513e9`::ref 0 aborting False >Thread-43::DEBUG::2014-01-22 16:38:57,156::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-43::DEBUG::2014-01-22 16:38:57,156::task::579::TaskManager.Task::(_updateState) Task=`481c6af0-54cb-4c7d-b800-b0a1462fd295`::moving from state init -> state preparing >Thread-43::INFO::2014-01-22 16:38:57,157::logUtils::44::dispatcher::(wrapper) Run and protect: spmStop(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-43::DEBUG::2014-01-22 16:38:57,158::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`c83e6a71-aa1d-4f88-8513-7baffefb5737`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '615' at 'spmStop' >Thread-43::DEBUG::2014-01-22 16:38:57,158::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-43::DEBUG::2014-01-22 16:38:57,158::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-43::DEBUG::2014-01-22 16:38:57,158::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`c83e6a71-aa1d-4f88-8513-7baffefb5737`::Granted request >Thread-43::DEBUG::2014-01-22 16:38:57,158::task::811::TaskManager.Task::(resourceAcquired) Task=`481c6af0-54cb-4c7d-b800-b0a1462fd295`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-43::DEBUG::2014-01-22 16:38:57,159::task::974::TaskManager.Task::(_decref) Task=`481c6af0-54cb-4c7d-b800-b0a1462fd295`::ref 1 aborting False >Thread-43::DEBUG::2014-01-22 16:38:57,159::sp::343::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process >Thread-43::DEBUG::2014-01-22 16:38:57,159::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`a563897e-ba19-4d96-be86-a256c7192dab`::Request was made in '/usr/share/vdsm/storage/sp.py' line '345' at '_shutDownUpgrade' >Thread-43::DEBUG::2014-01-22 16:38:57,159::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-43::DEBUG::2014-01-22 16:38:57,159::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-43::DEBUG::2014-01-22 16:38:57,159::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`a563897e-ba19-4d96-be86-a256c7192dab`::Granted request >Thread-43::DEBUG::2014-01-22 16:38:57,159::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-43::DEBUG::2014-01-22 16:38:57,159::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-43::DEBUG::2014-01-22 16:38:57,159::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-43::DEBUG::2014-01-22 16:38:57,159::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-43::DEBUG::2014-01-22 16:38:57,160::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-43::DEBUG::2014-01-22 16:38:57,160::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-43::DEBUG::2014-01-22 16:38:57,160::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-43::DEBUG::2014-01-22 16:38:57,165::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-43::INFO::2014-01-22 16:38:57,165::clusterlock::255::SANLock::(release) Releasing cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-43::DEBUG::2014-01-22 16:38:57,167::clusterlock::266::SANLock::(release) Cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully released >Thread-43::INFO::2014-01-22 16:38:57,167::logUtils::47::dispatcher::(wrapper) Run and protect: spmStop, Return response: None >Thread-43::DEBUG::2014-01-22 16:38:57,167::task::1168::TaskManager.Task::(prepare) Task=`481c6af0-54cb-4c7d-b800-b0a1462fd295`::finished: None >Thread-43::DEBUG::2014-01-22 16:38:57,168::task::579::TaskManager.Task::(_updateState) Task=`481c6af0-54cb-4c7d-b800-b0a1462fd295`::moving from state preparing -> state finished >Thread-43::DEBUG::2014-01-22 16:38:57,168::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-43::DEBUG::2014-01-22 16:38:57,168::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-43::DEBUG::2014-01-22 16:38:57,168::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-43::DEBUG::2014-01-22 16:38:57,169::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-43::DEBUG::2014-01-22 16:38:57,169::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-43::DEBUG::2014-01-22 16:38:57,169::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-43::DEBUG::2014-01-22 16:38:57,169::task::974::TaskManager.Task::(_decref) Task=`481c6af0-54cb-4c7d-b800-b0a1462fd295`::ref 0 aborting False >Thread-44::DEBUG::2014-01-22 16:38:57,221::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-44::DEBUG::2014-01-22 16:38:57,221::task::579::TaskManager.Task::(_updateState) Task=`e04baedf-be06-4035-941d-a8b87290750b`::moving from state init -> state preparing >Thread-44::INFO::2014-01-22 16:38:57,222::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', hostID=1, scsiKey='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', remove=False, options=None) >Thread-44::DEBUG::2014-01-22 16:38:57,222::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`e50d5f0d-bb57-47f2-b770-147c2db08713`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1111' at 'disconnectStoragePool' >Thread-44::DEBUG::2014-01-22 16:38:57,222::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-44::DEBUG::2014-01-22 16:38:57,223::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-44::DEBUG::2014-01-22 16:38:57,223::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`e50d5f0d-bb57-47f2-b770-147c2db08713`::Granted request >Thread-44::DEBUG::2014-01-22 16:38:57,223::task::811::TaskManager.Task::(resourceAcquired) Task=`e04baedf-be06-4035-941d-a8b87290750b`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-44::DEBUG::2014-01-22 16:38:57,223::task::974::TaskManager.Task::(_decref) Task=`e04baedf-be06-4035-941d-a8b87290750b`::ref 1 aborting False >Thread-44::DEBUG::2014-01-22 16:38:57,224::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`6c26d7ba-16bb-45b5-9360-aa0aace5be9d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1119' at '_disconnectPool' >Thread-44::DEBUG::2014-01-22 16:38:57,224::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-44::DEBUG::2014-01-22 16:38:57,224::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-44::DEBUG::2014-01-22 16:38:57,224::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`6c26d7ba-16bb-45b5-9360-aa0aace5be9d`::Granted request >Thread-44::INFO::2014-01-22 16:38:57,225::sp::711::Storage.StoragePool::(disconnect) Disconnect from the storage pool e1263145-a6a9-4993-9d8a-cd92a3e5c1fb >Thread-44::INFO::2014-01-22 16:38:57,225::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-27::DEBUG::2014-01-22 16:38:57,226::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-27::INFO::2014-01-22 16:38:57,226::clusterlock::197::SANLock::(releaseHostId) Releasing host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-27::DEBUG::2014-01-22 16:38:59,228::clusterlock::207::SANLock::(releaseHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 released successfully (id: 1) >Thread-44::DEBUG::2014-01-22 16:38:59,229::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-44::DEBUG::2014-01-22 16:38:59,229::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-44::DEBUG::2014-01-22 16:38:59,229::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-44::DEBUG::2014-01-22 16:38:59,229::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-44::INFO::2014-01-22 16:38:59,230::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: True >Thread-44::DEBUG::2014-01-22 16:38:59,230::task::1168::TaskManager.Task::(prepare) Task=`e04baedf-be06-4035-941d-a8b87290750b`::finished: True >Thread-44::DEBUG::2014-01-22 16:38:59,230::task::579::TaskManager.Task::(_updateState) Task=`e04baedf-be06-4035-941d-a8b87290750b`::moving from state preparing -> state finished >Thread-44::DEBUG::2014-01-22 16:38:59,231::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-44::DEBUG::2014-01-22 16:38:59,231::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-44::DEBUG::2014-01-22 16:38:59,231::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-44::DEBUG::2014-01-22 16:38:59,231::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-44::DEBUG::2014-01-22 16:38:59,232::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-44::DEBUG::2014-01-22 16:38:59,232::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-44::DEBUG::2014-01-22 16:38:59,232::task::974::TaskManager.Task::(_decref) Task=`e04baedf-be06-4035-941d-a8b87290750b`::ref 0 aborting False >Thread-45::DEBUG::2014-01-22 16:38:59,285::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-45::DEBUG::2014-01-22 16:38:59,286::task::579::TaskManager.Task::(_updateState) Task=`6c6b5a3e-c8fb-4121-8499-83ba588c610b`::moving from state init -> state preparing >Thread-45::INFO::2014-01-22 16:38:59,286::logUtils::44::dispatcher::(wrapper) Run and protect: startMonitoringDomain(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', hostID=1, options=None) >Thread-45::DEBUG::2014-01-22 16:38:59,287::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`d744d3fd-1102-483f-a606-44f7e49e588c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3694' at 'startMonitoringDomain' >Thread-45::DEBUG::2014-01-22 16:38:59,287::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-45::DEBUG::2014-01-22 16:38:59,287::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-45::DEBUG::2014-01-22 16:38:59,287::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`d744d3fd-1102-483f-a606-44f7e49e588c`::Granted request >Thread-45::INFO::2014-01-22 16:38:59,288::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-46::DEBUG::2014-01-22 16:38:59,288::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-45::DEBUG::2014-01-22 16:38:59,289::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-45::DEBUG::2014-01-22 16:38:59,289::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-45::DEBUG::2014-01-22 16:38:59,290::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-45::DEBUG::2014-01-22 16:38:59,291::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-45::INFO::2014-01-22 16:38:59,291::logUtils::47::dispatcher::(wrapper) Run and protect: startMonitoringDomain, Return response: None >Thread-45::DEBUG::2014-01-22 16:38:59,292::task::1168::TaskManager.Task::(prepare) Task=`6c6b5a3e-c8fb-4121-8499-83ba588c610b`::finished: None >Thread-45::DEBUG::2014-01-22 16:38:59,293::task::579::TaskManager.Task::(_updateState) Task=`6c6b5a3e-c8fb-4121-8499-83ba588c610b`::moving from state preparing -> state finished >Thread-45::DEBUG::2014-01-22 16:38:59,293::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-45::DEBUG::2014-01-22 16:38:59,294::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-45::DEBUG::2014-01-22 16:38:59,295::task::974::TaskManager.Task::(_decref) Task=`6c6b5a3e-c8fb-4121-8499-83ba588c610b`::ref 0 aborting False >Thread-46::DEBUG::2014-01-22 16:38:59,305::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-01-22 16:38:59,312::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000366841 s, 1.3 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-01-22 16:38:59,316::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3900e952-713e-4ea9-b32c-097cd4994bc5 changed its status to Valid >Thread-47::DEBUG::2014-01-22 16:38:59,317::misc::931::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-46::INFO::2014-01-22 16:38:59,317::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-47::DEBUG::2014-01-22 16:38:59,318::misc::941::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-46::DEBUG::2014-01-22 16:38:59,319::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >Thread-47::DEBUG::2014-01-22 16:38:59,320::misc::951::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-48::INFO::2014-01-22 16:38:59,324::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-49::DEBUG::2014-01-22 16:39:04,353::task::579::TaskManager.Task::(_updateState) Task=`d1bcb1b4-3c07-4d24-aef7-4cd2ab4bff1d`::moving from state init -> state preparing >Thread-49::INFO::2014-01-22 16:39:04,353::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-49::INFO::2014-01-22 16:39:04,353::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000366841', 'lastCheck': '5.0', 'valid': True}} >Thread-49::DEBUG::2014-01-22 16:39:04,354::task::1168::TaskManager.Task::(prepare) Task=`d1bcb1b4-3c07-4d24-aef7-4cd2ab4bff1d`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000366841', 'lastCheck': '5.0', 'valid': True}} >Thread-49::DEBUG::2014-01-22 16:39:04,354::task::579::TaskManager.Task::(_updateState) Task=`d1bcb1b4-3c07-4d24-aef7-4cd2ab4bff1d`::moving from state preparing -> state finished >Thread-49::DEBUG::2014-01-22 16:39:04,354::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-49::DEBUG::2014-01-22 16:39:04,354::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-49::DEBUG::2014-01-22 16:39:04,355::task::974::TaskManager.Task::(_decref) Task=`d1bcb1b4-3c07-4d24-aef7-4cd2ab4bff1d`::ref 0 aborting False >Thread-49::ERROR::2014-01-22 16:39:04,356::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 193, in get_local_host_score > self._config = config.Config() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 57, in __init__ > self._load(Config.static_files) > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 63, in _load > with open(fname, 'r') as f: >IOError: [Errno 2] No such file or directory: '/etc/ovirt-hosted-engine/hosted-engine.conf' >Thread-46::DEBUG::2014-01-22 16:39:09,330::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-01-22 16:39:09,338::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000321139 s, 1.5 MB/s\n'; <rc> = 0 >Thread-50::DEBUG::2014-01-22 16:39:09,422::task::579::TaskManager.Task::(_updateState) Task=`4174910a-4d38-4957-a1b4-aedc026e9a31`::moving from state init -> state preparing >Thread-50::INFO::2014-01-22 16:39:09,422::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-50::INFO::2014-01-22 16:39:09,422::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000321139', 'lastCheck': '0.1', 'valid': True}} >Thread-50::DEBUG::2014-01-22 16:39:09,423::task::1168::TaskManager.Task::(prepare) Task=`4174910a-4d38-4957-a1b4-aedc026e9a31`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000321139', 'lastCheck': '0.1', 'valid': True}} >Thread-50::DEBUG::2014-01-22 16:39:09,423::task::579::TaskManager.Task::(_updateState) Task=`4174910a-4d38-4957-a1b4-aedc026e9a31`::moving from state preparing -> state finished >Thread-50::DEBUG::2014-01-22 16:39:09,423::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-50::DEBUG::2014-01-22 16:39:09,423::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-50::DEBUG::2014-01-22 16:39:09,424::task::974::TaskManager.Task::(_decref) Task=`4174910a-4d38-4957-a1b4-aedc026e9a31`::ref 0 aborting False >Thread-50::ERROR::2014-01-22 16:39:09,425::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 193, in get_local_host_score > self._config = config.Config() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 57, in __init__ > self._load(Config.static_files) > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 63, in _load > with open(fname, 'r') as f: >IOError: [Errno 2] No such file or directory: '/etc/ovirt-hosted-engine/hosted-engine.conf' >Thread-51::DEBUG::2014-01-22 16:39:14,490::task::579::TaskManager.Task::(_updateState) Task=`96f56ec3-9f56-400a-af99-363d9f953976`::moving from state init -> state preparing >Thread-51::INFO::2014-01-22 16:39:14,490::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-51::INFO::2014-01-22 16:39:14,490::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000321139', 'lastCheck': '5.1', 'valid': True}} >Thread-51::DEBUG::2014-01-22 16:39:14,491::task::1168::TaskManager.Task::(prepare) Task=`96f56ec3-9f56-400a-af99-363d9f953976`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000321139', 'lastCheck': '5.1', 'valid': True}} >Thread-51::DEBUG::2014-01-22 16:39:14,491::task::579::TaskManager.Task::(_updateState) Task=`96f56ec3-9f56-400a-af99-363d9f953976`::moving from state preparing -> state finished >Thread-51::DEBUG::2014-01-22 16:39:14,491::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-51::DEBUG::2014-01-22 16:39:14,491::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-51::DEBUG::2014-01-22 16:39:14,492::task::974::TaskManager.Task::(_decref) Task=`96f56ec3-9f56-400a-af99-363d9f953976`::ref 0 aborting False >Thread-51::ERROR::2014-01-22 16:39:14,493::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 193, in get_local_host_score > self._config = config.Config() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 57, in __init__ > self._load(Config.static_files) > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 63, in _load > with open(fname, 'r') as f: >IOError: [Errno 2] No such file or directory: '/etc/ovirt-hosted-engine/hosted-engine.conf' >Thread-46::DEBUG::2014-01-22 16:39:19,352::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-01-22 16:39:19,360::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000354499 s, 1.4 MB/s\n'; <rc> = 0 >Thread-52::DEBUG::2014-01-22 16:39:19,558::task::579::TaskManager.Task::(_updateState) Task=`24c4ff5e-acea-4301-bff3-98412103469d`::moving from state init -> state preparing >Thread-52::INFO::2014-01-22 16:39:19,558::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-52::INFO::2014-01-22 16:39:19,558::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000354499', 'lastCheck': '0.2', 'valid': True}} >Thread-52::DEBUG::2014-01-22 16:39:19,558::task::1168::TaskManager.Task::(prepare) Task=`24c4ff5e-acea-4301-bff3-98412103469d`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000354499', 'lastCheck': '0.2', 'valid': True}} >Thread-52::DEBUG::2014-01-22 16:39:19,558::task::579::TaskManager.Task::(_updateState) Task=`24c4ff5e-acea-4301-bff3-98412103469d`::moving from state preparing -> state finished >Thread-52::DEBUG::2014-01-22 16:39:19,558::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-52::DEBUG::2014-01-22 16:39:19,558::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-52::DEBUG::2014-01-22 16:39:19,559::task::974::TaskManager.Task::(_decref) Task=`24c4ff5e-acea-4301-bff3-98412103469d`::ref 0 aborting False >Thread-52::ERROR::2014-01-22 16:39:19,559::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 193, in get_local_host_score > self._config = config.Config() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 57, in __init__ > self._load(Config.static_files) > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 63, in _load > with open(fname, 'r') as f: >IOError: [Errno 2] No such file or directory: '/etc/ovirt-hosted-engine/hosted-engine.conf' >Thread-53::DEBUG::2014-01-22 16:39:24,623::task::579::TaskManager.Task::(_updateState) Task=`3ebd669b-ee2e-4f97-9471-9e8b7bc73053`::moving from state init -> state preparing >Thread-53::INFO::2014-01-22 16:39:24,623::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-53::INFO::2014-01-22 16:39:24,623::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000354499', 'lastCheck': '5.3', 'valid': True}} >Thread-53::DEBUG::2014-01-22 16:39:24,623::task::1168::TaskManager.Task::(prepare) Task=`3ebd669b-ee2e-4f97-9471-9e8b7bc73053`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000354499', 'lastCheck': '5.3', 'valid': True}} >Thread-53::DEBUG::2014-01-22 16:39:24,624::task::579::TaskManager.Task::(_updateState) Task=`3ebd669b-ee2e-4f97-9471-9e8b7bc73053`::moving from state preparing -> state finished >Thread-53::DEBUG::2014-01-22 16:39:24,624::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-53::DEBUG::2014-01-22 16:39:24,624::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-53::DEBUG::2014-01-22 16:39:24,624::task::974::TaskManager.Task::(_decref) Task=`3ebd669b-ee2e-4f97-9471-9e8b7bc73053`::ref 0 aborting False >Thread-53::ERROR::2014-01-22 16:39:24,625::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 193, in get_local_host_score > self._config = config.Config() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 57, in __init__ > self._load(Config.static_files) > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 63, in _load > with open(fname, 'r') as f: >IOError: [Errno 2] No such file or directory: '/etc/ovirt-hosted-engine/hosted-engine.conf' >Thread-46::DEBUG::2014-01-22 16:39:29,375::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-01-22 16:39:29,382::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000357632 s, 1.3 MB/s\n'; <rc> = 0 >Thread-54::DEBUG::2014-01-22 16:39:29,690::task::579::TaskManager.Task::(_updateState) Task=`db523630-eb49-452b-a34e-32dda60d54d0`::moving from state init -> state preparing >Thread-54::INFO::2014-01-22 16:39:29,690::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-54::INFO::2014-01-22 16:39:29,690::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000357632', 'lastCheck': '0.3', 'valid': True}} >Thread-54::DEBUG::2014-01-22 16:39:29,691::task::1168::TaskManager.Task::(prepare) Task=`db523630-eb49-452b-a34e-32dda60d54d0`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000357632', 'lastCheck': '0.3', 'valid': True}} >Thread-54::DEBUG::2014-01-22 16:39:29,691::task::579::TaskManager.Task::(_updateState) Task=`db523630-eb49-452b-a34e-32dda60d54d0`::moving from state preparing -> state finished >Thread-54::DEBUG::2014-01-22 16:39:29,691::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-54::DEBUG::2014-01-22 16:39:29,691::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-54::DEBUG::2014-01-22 16:39:29,692::task::974::TaskManager.Task::(_decref) Task=`db523630-eb49-452b-a34e-32dda60d54d0`::ref 0 aborting False >Thread-54::ERROR::2014-01-22 16:39:29,693::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 193, in get_local_host_score > self._config = config.Config() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 57, in __init__ > self._load(Config.static_files) > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 63, in _load > with open(fname, 'r') as f: >IOError: [Errno 2] No such file or directory: '/etc/ovirt-hosted-engine/hosted-engine.conf' >Thread-55::DEBUG::2014-01-22 16:39:35,376::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('277d29a5-5802-4e2f-9852-ebb6e785262f',) {} >Thread-55::DEBUG::2014-01-22 16:39:35,377::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} >Thread-56::DEBUG::2014-01-22 16:39:35,508::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmCreate with ({'emulatedMachine': 'rhel6.5.0', 'vmId': '277d29a5-5802-4e2f-9852-ebb6e785262f', 'devices': [{'index': '2', 'iface': 'ide', 'specParams': {}, 'readonly': 'true', 'deviceId': '50aff99c-9a7b-4a98-8889-59a4a5dab99f', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'index': '0', 'iface': 'virtio', 'format': 'raw', 'optional': 'false', 'poolID': '00000000-0000-0000-0000-000000000000', 'volumeID': '150ad1c0-3297-47c5-b79d-2fe98d16e759', 'imageID': '6f59738c-caf5-4be8-98f4-27eacf7775a3', 'specParams': {}, 'readonly': 'false', 'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'deviceId': '6f59738c-caf5-4be8-98f4-27eacf7775a3', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'shared': 'exclusive', 'propagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:16:3e:2d:ec:3f', 'linkActive': 'true', 'network': 'rhevm', 'bootOrder': '1', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': 'a3b14666-7ec2-47b9-bddb-c1a6ef0483f6', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface'}, {'device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': 'e39cf709-51ca-443d-aaa6-72b678784ebc', 'alias': 'console0'}], 'smp': '2', 'memSize': '4096', 'cpuType': 'Westmere', 'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'vmName': 'HostedEngine', 'display': 'qxl'},) {} >Thread-56::INFO::2014-01-22 16:39:35,518::API::648::vds::(_getNetworkIp) network None: using 0 >Thread-56::INFO::2014-01-22 16:39:35,519::clientIF::394::vds::(createVm) vmContainerLock acquired by vm 277d29a5-5802-4e2f-9852-ebb6e785262f >Thread-56::DEBUG::2014-01-22 16:39:35,520::libvirtconnection::131::libvirtconnection::(get) trying to connect libvirt >Thread-56::DEBUG::2014-01-22 16:39:35,561::clientIF::407::vds::(createVm) Total desktops after creation of 277d29a5-5802-4e2f-9852-ebb6e785262f is 1 >Thread-57::DEBUG::2014-01-22 16:39:35,561::vm::2115::vm.Vm::(_startUnderlyingVm) vmId=`277d29a5-5802-4e2f-9852-ebb6e785262f`::Start >Thread-56::DEBUG::2014-01-22 16:39:35,562::BindingXMLRPC::991::vds::(wrapper) return vmCreate with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'nicModel': 'rtl8139,pv', 'vmId': '277d29a5-5802-4e2f-9852-ebb6e785262f', 'emulatedMachine': 'rhel6.5.0', 'displayIp': '0', 'pid': '0', 'devices': [{'index': '2', 'iface': 'ide', 'specParams': {}, 'readonly': 'true', 'deviceId': '50aff99c-9a7b-4a98-8889-59a4a5dab99f', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'index': '0', 'iface': 'virtio', 'format': 'raw', 'optional': 'false', 'poolID': '00000000-0000-0000-0000-000000000000', 'volumeID': '150ad1c0-3297-47c5-b79d-2fe98d16e759', 'imageID': '6f59738c-caf5-4be8-98f4-27eacf7775a3', 'specParams': {}, 'readonly': 'false', 'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'deviceId': '6f59738c-caf5-4be8-98f4-27eacf7775a3', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'shared': 'exclusive', 'propagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:16:3e:2d:ec:3f', 'linkActive': 'true', 'network': 'rhevm', 'bootOrder': '1', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': 'a3b14666-7ec2-47b9-bddb-c1a6ef0483f6', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface'}, {'device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': 'e39cf709-51ca-443d-aaa6-72b678784ebc', 'alias': 'console0'}], 'smp': '2', 'vmType': 'kvm', 'displaySecurePort': '-1', 'memSize': '4096', 'displayPort': '-1', 'cpuType': 'Westmere', 'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'clientIp': '', 'vmName': 'HostedEngine', 'display': 'qxl'}} >Thread-57::DEBUG::2014-01-22 16:39:35,562::vm::2119::vm.Vm::(_startUnderlyingVm) vmId=`277d29a5-5802-4e2f-9852-ebb6e785262f`::_ongoingCreations acquired >Thread-57::INFO::2014-01-22 16:39:35,563::vm::2969::vm.Vm::(_run) vmId=`277d29a5-5802-4e2f-9852-ebb6e785262f`::VM wrapper has started >Thread-57::DEBUG::2014-01-22 16:39:35,564::task::579::TaskManager.Task::(_updateState) Task=`84d04126-e7fb-40df-86c2-4f01215e1312`::moving from state init -> state preparing >Thread-57::INFO::2014-01-22 16:39:35,565::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='6f59738c-caf5-4be8-98f4-27eacf7775a3', volUUID='150ad1c0-3297-47c5-b79d-2fe98d16e759', options=None) >Thread-57::INFO::2014-01-22 16:39:35,570::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-57::DEBUG::2014-01-22 16:39:35,570::task::1168::TaskManager.Task::(prepare) Task=`84d04126-e7fb-40df-86c2-4f01215e1312`::finished: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-57::DEBUG::2014-01-22 16:39:35,570::task::579::TaskManager.Task::(_updateState) Task=`84d04126-e7fb-40df-86c2-4f01215e1312`::moving from state preparing -> state finished >Thread-57::DEBUG::2014-01-22 16:39:35,570::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-57::DEBUG::2014-01-22 16:39:35,570::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-57::DEBUG::2014-01-22 16:39:35,571::task::974::TaskManager.Task::(_decref) Task=`84d04126-e7fb-40df-86c2-4f01215e1312`::ref 0 aborting False >Thread-57::INFO::2014-01-22 16:39:35,571::clientIF::353::vds::(prepareVolumePath) prepared volume path: >Thread-57::DEBUG::2014-01-22 16:39:35,571::task::579::TaskManager.Task::(_updateState) Task=`caf700f7-1adc-47fa-9ff5-7605a961ae0c`::moving from state init -> state preparing >Thread-57::INFO::2014-01-22 16:39:35,572::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='6f59738c-caf5-4be8-98f4-27eacf7775a3', leafUUID='150ad1c0-3297-47c5-b79d-2fe98d16e759') >Thread-57::DEBUG::2014-01-22 16:39:35,572::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`f58c9ab5-c1b4-4c8d-82b6-929b66ca07e7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3256' at 'prepareImage' >Thread-57::DEBUG::2014-01-22 16:39:35,572::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'shared' >Thread-57::DEBUG::2014-01-22 16:39:35,573::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'shared' (1 active user) >Thread-57::DEBUG::2014-01-22 16:39:35,573::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`f58c9ab5-c1b4-4c8d-82b6-929b66ca07e7`::Granted request >Thread-57::DEBUG::2014-01-22 16:39:35,573::task::811::TaskManager.Task::(resourceAcquired) Task=`caf700f7-1adc-47fa-9ff5-7605a961ae0c`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (shared) >Thread-57::DEBUG::2014-01-22 16:39:35,574::task::974::TaskManager.Task::(_decref) Task=`caf700f7-1adc-47fa-9ff5-7605a961ae0c`::ref 1 aborting False >Thread-57::DEBUG::2014-01-22 16:39:35,577::fileVolume::528::Storage.Volume::(validateVolumePath) validate path for 150ad1c0-3297-47c5-b79d-2fe98d16e759 >Thread-57::INFO::2014-01-22 16:39:35,579::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759', 'volumeID': '150ad1c0-3297-47c5-b79d-2fe98d16e759', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759.lease', 'imageID': '6f59738c-caf5-4be8-98f4-27eacf7775a3'}, 'path': '/var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759', 'imgVolumesInfo': [{'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759', 'volumeID': '150ad1c0-3297-47c5-b79d-2fe98d16e759', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759.lease', 'imageID': '6f59738c-caf5-4be8-98f4-27eacf7775a3'}]} >Thread-57::DEBUG::2014-01-22 16:39:35,579::task::1168::TaskManager.Task::(prepare) Task=`caf700f7-1adc-47fa-9ff5-7605a961ae0c`::finished: {'info': {'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759', 'volumeID': '150ad1c0-3297-47c5-b79d-2fe98d16e759', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759.lease', 'imageID': '6f59738c-caf5-4be8-98f4-27eacf7775a3'}, 'path': '/var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759', 'imgVolumesInfo': [{'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759', 'volumeID': '150ad1c0-3297-47c5-b79d-2fe98d16e759', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759.lease', 'imageID': '6f59738c-caf5-4be8-98f4-27eacf7775a3'}]} >Thread-57::DEBUG::2014-01-22 16:39:35,579::task::579::TaskManager.Task::(_updateState) Task=`caf700f7-1adc-47fa-9ff5-7605a961ae0c`::moving from state preparing -> state finished >Thread-57::DEBUG::2014-01-22 16:39:35,579::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>} >Thread-57::DEBUG::2014-01-22 16:39:35,579::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-57::DEBUG::2014-01-22 16:39:35,580::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >Thread-57::DEBUG::2014-01-22 16:39:35,580::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >Thread-57::DEBUG::2014-01-22 16:39:35,580::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >Thread-57::DEBUG::2014-01-22 16:39:35,580::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >Thread-57::DEBUG::2014-01-22 16:39:35,580::task::974::TaskManager.Task::(_decref) Task=`caf700f7-1adc-47fa-9ff5-7605a961ae0c`::ref 0 aborting False >Thread-57::INFO::2014-01-22 16:39:35,581::clientIF::353::vds::(prepareVolumePath) prepared volume path: /var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759 >Thread-57::DEBUG::2014-01-22 16:39:35,595::utils::489::root::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_hostedengine' (cwd None) >Thread-57::DEBUG::2014-01-22 16:39:35,637::utils::509::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-57::INFO::2014-01-22 16:39:35,638::hooks::88::root::(_runHooksDir) >Thread-57::DEBUG::2014-01-22 16:39:35,638::vm::3031::vm.Vm::(_run) vmId=`277d29a5-5802-4e2f-9852-ebb6e785262f`::<?xml version="1.0" encoding="utf-8"?><domain type="kvm"> > <name>HostedEngine</name> > <uuid>277d29a5-5802-4e2f-9852-ebb6e785262f</uuid> > <memory>4194304</memory> > <currentMemory>4194304</currentMemory> > <vcpu>2</vcpu> > <memtune> > <min_guarantee>0</min_guarantee> > </memtune> > <devices> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/277d29a5-5802-4e2f-9852-ebb6e785262f.com.redhat.rhevm.vdsm"/> > </channel> > <channel type="unix"> > <target name="org.qemu.guest_agent.0" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/277d29a5-5802-4e2f-9852-ebb6e785262f.org.qemu.guest_agent.0"/> > </channel> > <input bus="ps2" type="mouse"/> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="display"/> > <channel mode="secure" name="inputs"/> > <channel mode="secure" name="cursor"/> > <channel mode="secure" name="playback"/> > <channel mode="secure" name="record"/> > <channel mode="secure" name="smartcard"/> > <channel mode="secure" name="usbredir"/> > </graphics> > <controller model="virtio-scsi" type="scsi"/> > <interface type="bridge"> > <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/> > <mac address="00:16:3e:2d:ec:3f"/> > <model type="virtio"/> > <source bridge="rhevm"/> > <filterref filter="vdsm-no-mac-spoofing"/> > <link state="up"/> > <boot order="1"/> > </interface> > <disk device="cdrom" snapshot="no" type="file"> > <address bus="1" controller="0" target="0" type="drive" unit="0"/> > <source file="" startupPolicy="optional"/> > <target bus="ide" dev="hdc"/> > <readonly/> > <serial/> > </disk> > <disk device="disk" snapshot="no" type="file"> > <address bus="0x00" domain="0x0000" function="0x0" slot="0x06" type="pci"/> > <source file="/var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759"/> > <target bus="virtio" dev="vda"/> > <serial>6f59738c-caf5-4be8-98f4-27eacf7775a3</serial> > <driver cache="none" error_policy="stop" io="threads" name="qemu" type="raw"/> > </disk> > <console type="pty"> > <target port="0" type="virtio"/> > </console> > <memballoon model="none"/> > <lease> > <key>150ad1c0-3297-47c5-b79d-2fe98d16e759</key> > <lockspace>3900e952-713e-4ea9-b32c-097cd4994bc5</lockspace> > <target offset="0" path="/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/6f59738c-caf5-4be8-98f4-27eacf7775a3/150ad1c0-3297-47c5-b79d-2fe98d16e759.lease"/> > </lease> > </devices> > <os> > <type arch="x86_64" machine="rhel6.5.0">hvm</type> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6Server-6.5.0.1.el6</entry> > <entry name="serial">4C4C4544-0034-5310-8052-B3C04F4A354A_d4:ae:52:c7:1a:39</entry> > <entry name="uuid">277d29a5-5802-4e2f-9852-ebb6e785262f</entry> > </system> > </sysinfo> > <clock adjustment="0" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > </cpu> ><on_poweroff>destroy</on_poweroff><on_reboot>destroy</on_reboot><on_crash>destroy</on_crash></domain> >Thread-59::DEBUG::2014-01-22 16:39:35,721::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-59::ERROR::2014-01-22 16:39:35,723::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-57::DEBUG::2014-01-22 16:39:36,457::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 1 edom: 10 level: 2 message: internal error process exited while connecting to monitor: char device redirected to /dev/pts/2 >((null):5247): Spice-Warning **: reds.c:3255:reds_init_ssl: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem >failed to initialize spice server >Thread-57::DEBUG::2014-01-22 16:39:36,457::vm::2136::vm.Vm::(_startUnderlyingVm) vmId=`277d29a5-5802-4e2f-9852-ebb6e785262f`::_ongoingCreations released >Thread-57::ERROR::2014-01-22 16:39:36,458::vm::2162::vm.Vm::(_startUnderlyingVm) vmId=`277d29a5-5802-4e2f-9852-ebb6e785262f`::The vm start process failed >Traceback (most recent call last): > File "/usr/share/vdsm/vm.py", line 2122, in _startUnderlyingVm > self._run() > File "/usr/share/vdsm/vm.py", line 3065, in _run > self._connection.createXML(domxml, flags), > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in createXML > if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) >libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/2 >((null):5247): Spice-Warning **: reds.c:3255:reds_init_ssl: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem >failed to initialize spice server >Thread-57::DEBUG::2014-01-22 16:39:36,469::vm::2615::vm.Vm::(setDownStatus) vmId=`277d29a5-5802-4e2f-9852-ebb6e785262f`::Changed state to Down: internal error process exited while connecting to monitor: char device redirected to /dev/pts/2 >((null):5247): Spice-Warning **: reds.c:3255:reds_init_ssl: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem >failed to initialize spice server >Thread-60::DEBUG::2014-01-22 16:39:36,882::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-60::ERROR::2014-01-22 16:39:36,882::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-61::DEBUG::2014-01-22 16:39:38,024::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-61::ERROR::2014-01-22 16:39:38,024::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-62::DEBUG::2014-01-22 16:39:39,172::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-62::ERROR::2014-01-22 16:39:39,173::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-46::DEBUG::2014-01-22 16:39:39,399::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-01-22 16:39:39,409::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000336156 s, 1.4 MB/s\n'; <rc> = 0 >Thread-63::DEBUG::2014-01-22 16:39:40,318::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-63::ERROR::2014-01-22 16:39:40,319::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-64::DEBUG::2014-01-22 16:39:41,466::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-64::ERROR::2014-01-22 16:39:41,466::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-65::DEBUG::2014-01-22 16:39:42,609::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-65::ERROR::2014-01-22 16:39:42,609::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-66::DEBUG::2014-01-22 16:39:43,751::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-66::ERROR::2014-01-22 16:39:43,751::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-67::DEBUG::2014-01-22 16:39:44,897::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-67::ERROR::2014-01-22 16:39:44,898::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-68::DEBUG::2014-01-22 16:39:46,037::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('277d29a5-5802-4e2f-9852-ebb6e785262f', '6014arEY', '10800', 'disconnect', {}) {} >Thread-68::ERROR::2014-01-22 16:39:46,037::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >Thread-69::DEBUG::2014-01-22 16:39:47,109::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-69::DEBUG::2014-01-22 16:39:47,110::task::579::TaskManager.Task::(_updateState) Task=`a2a72308-efcd-436e-93af-4626df24cd5d`::moving from state init -> state preparing >Thread-69::INFO::2014-01-22 16:39:47,110::logUtils::44::dispatcher::(wrapper) Run and protect: stopMonitoringDomain(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', options=None) >Thread-69::DEBUG::2014-01-22 16:39:47,111::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`816fcaae-2f98-4d41-a716-3867e198a201`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3701' at 'stopMonitoringDomain' >Thread-69::DEBUG::2014-01-22 16:39:47,111::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-69::DEBUG::2014-01-22 16:39:47,111::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-69::DEBUG::2014-01-22 16:39:47,111::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`816fcaae-2f98-4d41-a716-3867e198a201`::Granted request >Thread-69::INFO::2014-01-22 16:39:47,112::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-46::DEBUG::2014-01-22 16:39:47,112::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-46::INFO::2014-01-22 16:39:47,112::clusterlock::197::SANLock::(releaseHostId) Releasing host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-46::DEBUG::2014-01-22 16:39:49,113::clusterlock::207::SANLock::(releaseHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 released successfully (id: 1) >Thread-69::DEBUG::2014-01-22 16:39:49,114::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-69::DEBUG::2014-01-22 16:39:49,114::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-69::DEBUG::2014-01-22 16:39:49,115::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-69::DEBUG::2014-01-22 16:39:49,115::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-69::INFO::2014-01-22 16:39:49,115::logUtils::47::dispatcher::(wrapper) Run and protect: stopMonitoringDomain, Return response: None >Thread-69::DEBUG::2014-01-22 16:39:49,116::task::1168::TaskManager.Task::(prepare) Task=`a2a72308-efcd-436e-93af-4626df24cd5d`::finished: None >Thread-69::DEBUG::2014-01-22 16:39:49,116::task::579::TaskManager.Task::(_updateState) Task=`a2a72308-efcd-436e-93af-4626df24cd5d`::moving from state preparing -> state finished >Thread-69::DEBUG::2014-01-22 16:39:49,116::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69::DEBUG::2014-01-22 16:39:49,116::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69::DEBUG::2014-01-22 16:39:49,117::task::974::TaskManager.Task::(_decref) Task=`a2a72308-efcd-436e-93af-4626df24cd5d`::ref 0 aborting False >Thread-70::DEBUG::2014-01-22 16:43:24,422::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call getHardwareInfo with () {} >Thread-70::DEBUG::2014-01-22 16:43:24,438::BindingXMLRPC::991::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210 II', 'systemSerialNumber': '34SRJ5J', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0034-5310-8052-B3C04F4A354A', 'systemManufacturer': 'Dell Inc.'}} >Thread-71::DEBUG::2014-01-22 16:43:24,502::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-71::DEBUG::2014-01-22 16:43:24,503::task::579::TaskManager.Task::(_updateState) Task=`f4d9845d-cbff-437c-aeef-f31eba0225a6`::moving from state init -> state preparing >Thread-71::INFO::2014-01-22 16:43:24,503::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >Thread-71::INFO::2014-01-22 16:43:24,503::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >Thread-71::DEBUG::2014-01-22 16:43:24,503::task::1168::TaskManager.Task::(prepare) Task=`f4d9845d-cbff-437c-aeef-f31eba0225a6`::finished: {'poollist': []} >Thread-71::DEBUG::2014-01-22 16:43:24,504::task::579::TaskManager.Task::(_updateState) Task=`f4d9845d-cbff-437c-aeef-f31eba0225a6`::moving from state preparing -> state finished >Thread-71::DEBUG::2014-01-22 16:43:24,504::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71::DEBUG::2014-01-22 16:43:24,504::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71::DEBUG::2014-01-22 16:43:24,504::task::974::TaskManager.Task::(_decref) Task=`f4d9845d-cbff-437c-aeef-f31eba0225a6`::ref 0 aborting False >Thread-72::DEBUG::2014-01-22 16:43:32,246::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-72::DEBUG::2014-01-22 16:43:32,247::task::579::TaskManager.Task::(_updateState) Task=`6aa16681-59be-40ee-8679-13587a0de89c`::moving from state init -> state preparing >Thread-72::INFO::2014-01-22 16:43:32,247::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': '10.34.63.199:/jbelka/hosted', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': 'e3441208-c4f1-4f62-b2fb-fbcf986d3c0d'}], options=None) >Thread-72::DEBUG::2014-01-22 16:43:32,256::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted >Thread-72::DEBUG::2014-01-22 16:43:32,258::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('3900e952-713e-4ea9-b32c-097cd4994bc5',) >Thread-72::DEBUG::2014-01-22 16:43:32,258::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {3900e952-713e-4ea9-b32c-097cd4994bc5: storage.nfsSD.findDomain} >Thread-72::INFO::2014-01-22 16:43:32,258::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'e3441208-c4f1-4f62-b2fb-fbcf986d3c0d'}]} >Thread-72::DEBUG::2014-01-22 16:43:32,258::task::1168::TaskManager.Task::(prepare) Task=`6aa16681-59be-40ee-8679-13587a0de89c`::finished: {'statuslist': [{'status': 0, 'id': 'e3441208-c4f1-4f62-b2fb-fbcf986d3c0d'}]} >Thread-72::DEBUG::2014-01-22 16:43:32,259::task::579::TaskManager.Task::(_updateState) Task=`6aa16681-59be-40ee-8679-13587a0de89c`::moving from state preparing -> state finished >Thread-72::DEBUG::2014-01-22 16:43:32,259::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72::DEBUG::2014-01-22 16:43:32,259::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72::DEBUG::2014-01-22 16:43:32,259::task::974::TaskManager.Task::(_decref) Task=`6aa16681-59be-40ee-8679-13587a0de89c`::ref 0 aborting False >Thread-73::DEBUG::2014-01-22 16:43:32,310::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-73::DEBUG::2014-01-22 16:43:32,310::task::579::TaskManager.Task::(_updateState) Task=`2aa8f572-9ac7-48f0-a51c-a474a96139dd`::moving from state init -> state preparing >Thread-73::INFO::2014-01-22 16:43:32,311::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainsList(spUUID='00000000-0000-0000-0000-000000000000', domainClass=None, storageType=None, remotePath=None, options=None) >Thread-73::DEBUG::2014-01-22 16:43:32,311::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-73::DEBUG::2014-01-22 16:43:32,311::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-73::DEBUG::2014-01-22 16:43:32,311::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-73::DEBUG::2014-01-22 16:43:32,311::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-73::DEBUG::2014-01-22 16:43:32,312::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-73::DEBUG::2014-01-22 16:43:32,340::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-73::DEBUG::2014-01-22 16:43:32,340::misc::819::SamplingMethod::(__call__) Returning last result >Thread-73::DEBUG::2014-01-22 16:43:34,374::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-73::DEBUG::2014-01-22 16:43:34,487::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-73::DEBUG::2014-01-22 16:43:34,488::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-73::DEBUG::2014-01-22 16:43:34,488::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-73::DEBUG::2014-01-22 16:43:34,489::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-73::DEBUG::2014-01-22 16:43:34,489::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-73::DEBUG::2014-01-22 16:43:34,489::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-73::DEBUG::2014-01-22 16:43:34,490::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-73::DEBUG::2014-01-22 16:43:34,490::misc::819::SamplingMethod::(__call__) Returning last result >Thread-73::DEBUG::2014-01-22 16:43:34,490::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-73::DEBUG::2014-01-22 16:43:34,493::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-73::DEBUG::2014-01-22 16:43:34,635::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >Thread-73::DEBUG::2014-01-22 16:43:34,636::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-73::DEBUG::2014-01-22 16:43:34,654::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-73::DEBUG::2014-01-22 16:43:34,655::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-73::DEBUG::2014-01-22 16:43:34,664::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-73::DEBUG::2014-01-22 16:43:34,665::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-73::INFO::2014-01-22 16:43:34,666::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS already registered >Thread-73::INFO::2014-01-22 16:43:34,666::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS already registered >Thread-73::INFO::2014-01-22 16:43:34,666::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainsList, Return response: {'domlist': ['3900e952-713e-4ea9-b32c-097cd4994bc5']} >Thread-73::DEBUG::2014-01-22 16:43:34,666::task::1168::TaskManager.Task::(prepare) Task=`2aa8f572-9ac7-48f0-a51c-a474a96139dd`::finished: {'domlist': ['3900e952-713e-4ea9-b32c-097cd4994bc5']} >Thread-73::DEBUG::2014-01-22 16:43:34,666::task::579::TaskManager.Task::(_updateState) Task=`2aa8f572-9ac7-48f0-a51c-a474a96139dd`::moving from state preparing -> state finished >Thread-73::DEBUG::2014-01-22 16:43:34,666::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-73::DEBUG::2014-01-22 16:43:34,666::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-73::DEBUG::2014-01-22 16:43:34,666::task::974::TaskManager.Task::(_decref) Task=`2aa8f572-9ac7-48f0-a51c-a474a96139dd`::ref 0 aborting False >Thread-76::DEBUG::2014-01-22 16:43:34,735::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-76::DEBUG::2014-01-22 16:43:34,736::task::579::TaskManager.Task::(_updateState) Task=`1b153b11-7bd4-4f77-851f-73059b2b0c24`::moving from state init -> state preparing >Thread-76::INFO::2014-01-22 16:43:34,736::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainInfo(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', options=None) >Thread-76::INFO::2014-01-22 16:43:34,736::fileSD::299::Storage.StorageDomain::(validate) sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-76::DEBUG::2014-01-22 16:43:34,744::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-76::DEBUG::2014-01-22 16:43:34,745::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`aaf98530-420e-42a8-9c47-3dc78dc4897b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2759' at 'getStorageDomainInfo' >Thread-76::DEBUG::2014-01-22 16:43:34,745::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'shared' >Thread-76::DEBUG::2014-01-22 16:43:34,746::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'shared' (1 active user) >Thread-76::DEBUG::2014-01-22 16:43:34,746::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`aaf98530-420e-42a8-9c47-3dc78dc4897b`::Granted request >Thread-76::DEBUG::2014-01-22 16:43:34,746::task::811::TaskManager.Task::(resourceAcquired) Task=`1b153b11-7bd4-4f77-851f-73059b2b0c24`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (shared) >Thread-76::DEBUG::2014-01-22 16:43:34,747::task::974::TaskManager.Task::(_decref) Task=`1b153b11-7bd4-4f77-851f-73059b2b0c24`::ref 1 aborting False >Thread-76::INFO::2014-01-22 16:43:34,750::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainInfo, Return response: {'info': {'uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'master_ver': 0, 'lver': -1, 'version': '3', 'role': 'Master', 'remotePath': '10.34.63.199:/jbelka/hosted', 'spm_id': -1, 'type': 'NFS', 'class': 'Data', 'pool': ['e1263145-a6a9-4993-9d8a-cd92a3e5c1fb'], 'name': 'hosted_storage'}} >Thread-76::DEBUG::2014-01-22 16:43:34,750::task::1168::TaskManager.Task::(prepare) Task=`1b153b11-7bd4-4f77-851f-73059b2b0c24`::finished: {'info': {'uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'master_ver': 0, 'lver': -1, 'version': '3', 'role': 'Master', 'remotePath': '10.34.63.199:/jbelka/hosted', 'spm_id': -1, 'type': 'NFS', 'class': 'Data', 'pool': ['e1263145-a6a9-4993-9d8a-cd92a3e5c1fb'], 'name': 'hosted_storage'}} >Thread-76::DEBUG::2014-01-22 16:43:34,751::task::579::TaskManager.Task::(_updateState) Task=`1b153b11-7bd4-4f77-851f-73059b2b0c24`::moving from state preparing -> state finished >Thread-76::DEBUG::2014-01-22 16:43:34,751::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>} >Thread-76::DEBUG::2014-01-22 16:43:34,751::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-76::DEBUG::2014-01-22 16:43:34,751::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >Thread-76::DEBUG::2014-01-22 16:43:34,752::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >Thread-76::DEBUG::2014-01-22 16:43:34,752::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >Thread-76::DEBUG::2014-01-22 16:43:34,752::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >Thread-76::DEBUG::2014-01-22 16:43:34,752::task::974::TaskManager.Task::(_decref) Task=`1b153b11-7bd4-4f77-851f-73059b2b0c24`::ref 0 aborting False >Thread-77::DEBUG::2014-01-22 16:44:33,223::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-77::DEBUG::2014-01-22 16:44:33,225::task::579::TaskManager.Task::(_updateState) Task=`0beeb56f-4fef-4cb8-9835-ce7a3159000b`::moving from state init -> state preparing >Thread-77::INFO::2014-01-22 16:44:33,225::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', hostID=1, scsiKey='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', msdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', masterVersion=1, options=None) >Thread-77::DEBUG::2014-01-22 16:44:33,226::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`18ac70ae-2243-4bc2-b501-4281241071bb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool' >Thread-77::DEBUG::2014-01-22 16:44:33,226::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-77::DEBUG::2014-01-22 16:44:33,227::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-77::DEBUG::2014-01-22 16:44:33,227::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`18ac70ae-2243-4bc2-b501-4281241071bb`::Granted request >Thread-77::DEBUG::2014-01-22 16:44:33,227::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`739e76a7-0008-4785-a9aa-970c32b71c12`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool' >Thread-77::DEBUG::2014-01-22 16:44:33,228::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-77::DEBUG::2014-01-22 16:44:33,228::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-77::DEBUG::2014-01-22 16:44:33,228::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`739e76a7-0008-4785-a9aa-970c32b71c12`::Granted request >Thread-77::INFO::2014-01-22 16:44:33,228::sp::674::Storage.StoragePool::(connect) Connect host #1 to the storage pool e1263145-a6a9-4993-9d8a-cd92a3e5c1fb with master domain: 3900e952-713e-4ea9-b32c-097cd4994bc5 (ver = 1) >Thread-77::DEBUG::2014-01-22 16:44:33,228::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:33,229::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:33,229::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:33,229::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:33,229::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:33,229::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:33,229::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-77::DEBUG::2014-01-22 16:44:33,229::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-77::DEBUG::2014-01-22 16:44:33,229::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-77::DEBUG::2014-01-22 16:44:33,229::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-77::DEBUG::2014-01-22 16:44:33,229::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-77::DEBUG::2014-01-22 16:44:33,242::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-77::DEBUG::2014-01-22 16:44:33,243::misc::819::SamplingMethod::(__call__) Returning last result >Thread-77::DEBUG::2014-01-22 16:44:39,289::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-77::DEBUG::2014-01-22 16:44:39,369::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-77::DEBUG::2014-01-22 16:44:39,370::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:39,370::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:39,370::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:39,371::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:39,371::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:39,371::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:39,371::misc::819::SamplingMethod::(__call__) Returning last result >Thread-77::DEBUG::2014-01-22 16:44:39,378::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-77::DEBUG::2014-01-22 16:44:39,379::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-77::DEBUG::2014-01-22 16:44:39,387::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-77::DEBUG::2014-01-22 16:44:39,389::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-77::INFO::2014-01-22 16:44:39,389::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS already registered >Thread-77::INFO::2014-01-22 16:44:39,390::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS already registered >Thread-77::DEBUG::2014-01-22 16:44:39,390::sp::1529::Storage.StoragePool::(getMasterDomain) Master domain 3900e952-713e-4ea9-b32c-097cd4994bc5 verified, version 1 >Thread-77::DEBUG::2014-01-22 16:44:39,390::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-77::DEBUG::2014-01-22 16:44:39,390::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-77::DEBUG::2014-01-22 16:44:39,398::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-77::INFO::2014-01-22 16:44:39,398::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-77::DEBUG::2014-01-22 16:44:39,399::sp::1562::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `e1263145-a6a9-4993-9d8a-cd92a3e5c1fb` started monitoring domain `3900e952-713e-4ea9-b32c-097cd4994bc5` >Thread-79::DEBUG::2014-01-22 16:44:39,399::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-77::DEBUG::2014-01-22 16:44:39,400::misc::819::SamplingMethod::(__call__) Returning last result >Thread-77::DEBUG::2014-01-22 16:44:39,403::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-77::DEBUG::2014-01-22 16:44:39,413::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-79::DEBUG::2014-01-22 16:44:39,429::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-79::DEBUG::2014-01-22 16:44:39,436::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000459394 s, 1.0 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-01-22 16:44:39,441::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3900e952-713e-4ea9-b32c-097cd4994bc5 changed its status to Valid >Thread-80::DEBUG::2014-01-22 16:44:39,441::misc::931::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-79::INFO::2014-01-22 16:44:39,441::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-80::DEBUG::2014-01-22 16:44:39,442::misc::941::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-79::DEBUG::2014-01-22 16:44:39,443::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >Thread-80::DEBUG::2014-01-22 16:44:39,444::misc::951::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-81::INFO::2014-01-22 16:44:39,458::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-77::DEBUG::2014-01-22 16:44:39,513::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >Thread-77::DEBUG::2014-01-22 16:44:39,513::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-77::INFO::2014-01-22 16:44:39,516::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-77::INFO::2014-01-22 16:44:39,516::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd >Thread-77::DEBUG::2014-01-22 16:44:39,517::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-77::DEBUG::2014-01-22 16:44:39,517::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-77::DEBUG::2014-01-22 16:44:39,518::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-77::DEBUG::2014-01-22 16:44:39,518::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-77::DEBUG::2014-01-22 16:44:39,518::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-77::DEBUG::2014-01-22 16:44:39,518::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-77::DEBUG::2014-01-22 16:44:39,519::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-77::DEBUG::2014-01-22 16:44:39,519::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-77::INFO::2014-01-22 16:44:39,519::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-77::DEBUG::2014-01-22 16:44:39,519::task::1168::TaskManager.Task::(prepare) Task=`0beeb56f-4fef-4cb8-9835-ce7a3159000b`::finished: True >Thread-77::DEBUG::2014-01-22 16:44:39,519::task::579::TaskManager.Task::(_updateState) Task=`0beeb56f-4fef-4cb8-9835-ce7a3159000b`::moving from state preparing -> state finished >Thread-77::DEBUG::2014-01-22 16:44:39,519::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-77::DEBUG::2014-01-22 16:44:39,519::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-77::DEBUG::2014-01-22 16:44:39,520::task::974::TaskManager.Task::(_decref) Task=`0beeb56f-4fef-4cb8-9835-ce7a3159000b`::ref 0 aborting False >Thread-83::DEBUG::2014-01-22 16:44:39,569::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-83::DEBUG::2014-01-22 16:44:39,569::task::579::TaskManager.Task::(_updateState) Task=`cf18df00-35c8-45d6-a14b-1da978d1d08e`::moving from state init -> state preparing >Thread-83::INFO::2014-01-22 16:44:39,570::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-83::DEBUG::2014-01-22 16:44:39,570::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`5750676b-0fca-441a-906c-150aa0843e2f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2562' at 'getStoragePoolInfo' >Thread-83::DEBUG::2014-01-22 16:44:39,571::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'shared' >Thread-83::DEBUG::2014-01-22 16:44:39,571::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'shared' (1 active user) >Thread-83::DEBUG::2014-01-22 16:44:39,571::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`5750676b-0fca-441a-906c-150aa0843e2f`::Granted request >Thread-83::DEBUG::2014-01-22 16:44:39,572::task::811::TaskManager.Task::(resourceAcquired) Task=`cf18df00-35c8-45d6-a14b-1da978d1d08e`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (shared) >Thread-83::DEBUG::2014-01-22 16:44:39,572::task::974::TaskManager.Task::(_decref) Task=`cf18df00-35c8-45d6-a14b-1da978d1d08e`::ref 1 aborting False >Thread-83::INFO::2014-01-22 16:44:39,577::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': -1, 'master_uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'name': 'hosted_datacenter', 'version': '3', 'domains': '3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 0}, 'dominfo': {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'status': 'Active', 'diskfree': '75295424512', 'isoprefix': '', 'alerts': [], 'disktotal': '422943129600', 'version': 3}}} >Thread-83::DEBUG::2014-01-22 16:44:39,577::task::1168::TaskManager.Task::(prepare) Task=`cf18df00-35c8-45d6-a14b-1da978d1d08e`::finished: {'info': {'spm_id': -1, 'master_uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'name': 'hosted_datacenter', 'version': '3', 'domains': '3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 0}, 'dominfo': {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'status': 'Active', 'diskfree': '75295424512', 'isoprefix': '', 'alerts': [], 'disktotal': '422943129600', 'version': 3}}} >Thread-83::DEBUG::2014-01-22 16:44:39,577::task::579::TaskManager.Task::(_updateState) Task=`cf18df00-35c8-45d6-a14b-1da978d1d08e`::moving from state preparing -> state finished >Thread-83::DEBUG::2014-01-22 16:44:39,577::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-83::DEBUG::2014-01-22 16:44:39,578::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-83::DEBUG::2014-01-22 16:44:39,578::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-83::DEBUG::2014-01-22 16:44:39,578::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-83::DEBUG::2014-01-22 16:44:39,578::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-83::DEBUG::2014-01-22 16:44:39,578::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-83::DEBUG::2014-01-22 16:44:39,578::task::974::TaskManager.Task::(_decref) Task=`cf18df00-35c8-45d6-a14b-1da978d1d08e`::ref 0 aborting False >Thread-79::DEBUG::2014-01-22 16:44:49,457::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-79::DEBUG::2014-01-22 16:44:49,464::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000420072 s, 1.1 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-01-22 16:44:59,484::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-79::DEBUG::2014-01-22 16:44:59,491::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000337052 s, 1.4 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-01-22 16:45:09,504::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-79::DEBUG::2014-01-22 16:45:09,511::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000382323 s, 1.3 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-01-22 16:45:19,524::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-79::DEBUG::2014-01-22 16:45:19,531::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.00035662 s, 1.3 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-01-22 16:45:29,544::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-79::DEBUG::2014-01-22 16:45:29,551::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000344752 s, 1.4 MB/s\n'; <rc> = 0 >Thread-84::DEBUG::2014-01-22 16:45:33,614::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-84::DEBUG::2014-01-22 16:45:33,615::task::579::TaskManager.Task::(_updateState) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::moving from state init -> state preparing >Thread-84::INFO::2014-01-22 16:45:33,615::logUtils::44::dispatcher::(wrapper) Run and protect: spmStop(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-84::DEBUG::2014-01-22 16:45:33,616::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`9c8a65ef-0a3c-4f80-a115-2b7850156b9d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '615' at 'spmStop' >Thread-84::DEBUG::2014-01-22 16:45:33,616::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-84::DEBUG::2014-01-22 16:45:33,616::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-84::DEBUG::2014-01-22 16:45:33,617::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`9c8a65ef-0a3c-4f80-a115-2b7850156b9d`::Granted request >Thread-84::DEBUG::2014-01-22 16:45:33,617::task::811::TaskManager.Task::(resourceAcquired) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-84::DEBUG::2014-01-22 16:45:33,617::task::974::TaskManager.Task::(_decref) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::ref 1 aborting False >Thread-84::ERROR::2014-01-22 16:45:33,618::task::850::TaskManager.Task::(_setError) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 857, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 618, in spmStop > pool.stopSpm() > File "/usr/share/vdsm/storage/securable.py", line 66, in wrapper > raise SecureError() >SecureError >Thread-84::DEBUG::2014-01-22 16:45:33,619::task::869::TaskManager.Task::(_run) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::Task._run: b4b5b4b5-25be-4bac-b034-453fd0491e23 ('e1263145-a6a9-4993-9d8a-cd92a3e5c1fb',) {} failed - stopping task >Thread-84::DEBUG::2014-01-22 16:45:33,619::task::1194::TaskManager.Task::(stop) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::stopping in state preparing (force False) >Thread-84::DEBUG::2014-01-22 16:45:33,619::task::974::TaskManager.Task::(_decref) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::ref 1 aborting True >Thread-84::INFO::2014-01-22 16:45:33,619::task::1151::TaskManager.Task::(prepare) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::aborting: Task is aborted: u'' - code 100 >Thread-84::DEBUG::2014-01-22 16:45:33,619::task::1156::TaskManager.Task::(prepare) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::Prepare: aborted: >Thread-84::DEBUG::2014-01-22 16:45:33,619::task::974::TaskManager.Task::(_decref) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::ref 0 aborting True >Thread-84::DEBUG::2014-01-22 16:45:33,619::task::909::TaskManager.Task::(_doAbort) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::Task._doAbort: force False >Thread-84::DEBUG::2014-01-22 16:45:33,619::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-84::DEBUG::2014-01-22 16:45:33,620::task::579::TaskManager.Task::(_updateState) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::moving from state preparing -> state aborting >Thread-84::DEBUG::2014-01-22 16:45:33,620::task::534::TaskManager.Task::(__state_aborting) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::_aborting: recover policy none >Thread-84::DEBUG::2014-01-22 16:45:33,620::task::579::TaskManager.Task::(_updateState) Task=`b4b5b4b5-25be-4bac-b034-453fd0491e23`::moving from state aborting -> state failed >Thread-84::DEBUG::2014-01-22 16:45:33,620::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-84::DEBUG::2014-01-22 16:45:33,620::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-84::DEBUG::2014-01-22 16:45:33,620::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-84::DEBUG::2014-01-22 16:45:33,620::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-84::DEBUG::2014-01-22 16:45:33,620::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-84::DEBUG::2014-01-22 16:45:33,620::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-84::ERROR::2014-01-22 16:45:33,620::dispatcher::70::Storage.Dispatcher.Protect::(run) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 1159, in prepare > raise self.error >SecureError >Thread-85::DEBUG::2014-01-22 16:45:33,671::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-85::DEBUG::2014-01-22 16:45:33,671::task::579::TaskManager.Task::(_updateState) Task=`8f9393eb-e7cc-4c96-9a05-2ca0ced98332`::moving from state init -> state preparing >Thread-85::INFO::2014-01-22 16:45:33,672::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', hostID=1, scsiKey='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', remove=False, options=None) >Thread-85::DEBUG::2014-01-22 16:45:33,672::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`7be2f869-68ca-4def-8ce3-23eb2ada85ed`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1111' at 'disconnectStoragePool' >Thread-85::DEBUG::2014-01-22 16:45:33,672::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-85::DEBUG::2014-01-22 16:45:33,673::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-85::DEBUG::2014-01-22 16:45:33,673::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`7be2f869-68ca-4def-8ce3-23eb2ada85ed`::Granted request >Thread-85::DEBUG::2014-01-22 16:45:33,673::task::811::TaskManager.Task::(resourceAcquired) Task=`8f9393eb-e7cc-4c96-9a05-2ca0ced98332`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-85::DEBUG::2014-01-22 16:45:33,674::task::974::TaskManager.Task::(_decref) Task=`8f9393eb-e7cc-4c96-9a05-2ca0ced98332`::ref 1 aborting False >Thread-85::DEBUG::2014-01-22 16:45:33,674::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`c9518849-39ed-4d4b-b6a9-a7240fa68ac7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1119' at '_disconnectPool' >Thread-85::DEBUG::2014-01-22 16:45:33,674::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-85::DEBUG::2014-01-22 16:45:33,674::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-85::DEBUG::2014-01-22 16:45:33,675::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`c9518849-39ed-4d4b-b6a9-a7240fa68ac7`::Granted request >Thread-85::INFO::2014-01-22 16:45:33,675::sp::711::Storage.StoragePool::(disconnect) Disconnect from the storage pool e1263145-a6a9-4993-9d8a-cd92a3e5c1fb >Thread-85::INFO::2014-01-22 16:45:33,676::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-79::DEBUG::2014-01-22 16:45:33,676::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-79::INFO::2014-01-22 16:45:33,676::clusterlock::197::SANLock::(releaseHostId) Releasing host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-79::DEBUG::2014-01-22 16:45:35,677::clusterlock::207::SANLock::(releaseHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 released successfully (id: 1) >Thread-85::DEBUG::2014-01-22 16:45:35,678::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-85::DEBUG::2014-01-22 16:45:35,678::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-85::DEBUG::2014-01-22 16:45:35,678::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-85::DEBUG::2014-01-22 16:45:35,678::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-85::INFO::2014-01-22 16:45:35,679::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: True >Thread-85::DEBUG::2014-01-22 16:45:35,679::task::1168::TaskManager.Task::(prepare) Task=`8f9393eb-e7cc-4c96-9a05-2ca0ced98332`::finished: True >Thread-85::DEBUG::2014-01-22 16:45:35,679::task::579::TaskManager.Task::(_updateState) Task=`8f9393eb-e7cc-4c96-9a05-2ca0ced98332`::moving from state preparing -> state finished >Thread-85::DEBUG::2014-01-22 16:45:35,680::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-85::DEBUG::2014-01-22 16:45:35,680::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-85::DEBUG::2014-01-22 16:45:35,680::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-85::DEBUG::2014-01-22 16:45:35,681::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-85::DEBUG::2014-01-22 16:45:35,681::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-85::DEBUG::2014-01-22 16:45:35,681::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-85::DEBUG::2014-01-22 16:45:35,681::task::974::TaskManager.Task::(_decref) Task=`8f9393eb-e7cc-4c96-9a05-2ca0ced98332`::ref 0 aborting False >Thread-85::ERROR::2014-01-22 16:45:35,682::SecureXMLRPCServer::168::root::(handle_error) client ('127.0.0.1', 46817) >Traceback (most recent call last): > File "/usr/lib64/python2.6/SocketServer.py", line 570, in process_request_thread > self.finish_request(request, client_address) > File "/usr/lib64/python2.6/SocketServer.py", line 332, in finish_request > self.RequestHandlerClass(request, client_address, self) > File "/usr/lib64/python2.6/SocketServer.py", line 627, in __init__ > self.handle() > File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle > self.handle_one_request() > File "/usr/lib64/python2.6/BaseHTTPServer.py", line 323, in handle_one_request > method() > File "/usr/lib64/python2.6/SimpleXMLRPCServer.py", line 490, in do_POST > self.send_response(200) > File "/usr/lib64/python2.6/BaseHTTPServer.py", line 384, in send_response > self.send_header('Server', self.version_string()) > File "/usr/lib64/python2.6/BaseHTTPServer.py", line 390, in send_header > self.wfile.write("%s: %s\r\n" % (keyword, value)) > File "/usr/lib64/python2.6/socket.py", line 324, in write > self.flush() > File "/usr/lib64/python2.6/socket.py", line 303, in flush > self._sock.sendall(buffer(data, write_offset, buffer_size)) > File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 222, in write > return self._write_bio(data) > File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 205, in _write_bio > return m2.ssl_write(self.ssl, data, self._timeout) >SSLError: (32, 'Broken pipe') >Thread-86::DEBUG::2014-01-22 16:48:55,903::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call getHardwareInfo with () {} >Thread-86::DEBUG::2014-01-22 16:48:55,911::BindingXMLRPC::991::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210 II', 'systemSerialNumber': '34SRJ5J', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0034-5310-8052-B3C04F4A354A', 'systemManufacturer': 'Dell Inc.'}} >Thread-87::DEBUG::2014-01-22 16:48:55,975::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-87::DEBUG::2014-01-22 16:48:55,976::task::579::TaskManager.Task::(_updateState) Task=`eec8dec8-876d-4981-956c-6e66ffe6f8a5`::moving from state init -> state preparing >Thread-87::INFO::2014-01-22 16:48:55,976::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >Thread-87::INFO::2014-01-22 16:48:55,976::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >Thread-87::DEBUG::2014-01-22 16:48:55,977::task::1168::TaskManager.Task::(prepare) Task=`eec8dec8-876d-4981-956c-6e66ffe6f8a5`::finished: {'poollist': []} >Thread-87::DEBUG::2014-01-22 16:48:55,977::task::579::TaskManager.Task::(_updateState) Task=`eec8dec8-876d-4981-956c-6e66ffe6f8a5`::moving from state preparing -> state finished >Thread-87::DEBUG::2014-01-22 16:48:55,977::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-87::DEBUG::2014-01-22 16:48:55,977::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-87::DEBUG::2014-01-22 16:48:55,977::task::974::TaskManager.Task::(_decref) Task=`eec8dec8-876d-4981-956c-6e66ffe6f8a5`::ref 0 aborting False >Thread-88::DEBUG::2014-01-22 16:49:04,043::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-88::DEBUG::2014-01-22 16:49:04,043::task::579::TaskManager.Task::(_updateState) Task=`67ad0807-ff01-46ae-a200-1332dae43706`::moving from state init -> state preparing >Thread-88::INFO::2014-01-22 16:49:04,044::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': '10.34.63.199:/jbelka/hosted', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '9e10ccab-8a15-48c2-b1cc-49511bf53a28'}], options=None) >Thread-88::DEBUG::2014-01-22 16:49:04,047::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted >Thread-88::DEBUG::2014-01-22 16:49:04,052::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('3900e952-713e-4ea9-b32c-097cd4994bc5',) >Thread-88::DEBUG::2014-01-22 16:49:04,052::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {3900e952-713e-4ea9-b32c-097cd4994bc5: storage.nfsSD.findDomain} >Thread-88::INFO::2014-01-22 16:49:04,052::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '9e10ccab-8a15-48c2-b1cc-49511bf53a28'}]} >Thread-88::DEBUG::2014-01-22 16:49:04,052::task::1168::TaskManager.Task::(prepare) Task=`67ad0807-ff01-46ae-a200-1332dae43706`::finished: {'statuslist': [{'status': 0, 'id': '9e10ccab-8a15-48c2-b1cc-49511bf53a28'}]} >Thread-88::DEBUG::2014-01-22 16:49:04,053::task::579::TaskManager.Task::(_updateState) Task=`67ad0807-ff01-46ae-a200-1332dae43706`::moving from state preparing -> state finished >Thread-88::DEBUG::2014-01-22 16:49:04,053::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-88::DEBUG::2014-01-22 16:49:04,053::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-88::DEBUG::2014-01-22 16:49:04,053::task::974::TaskManager.Task::(_decref) Task=`67ad0807-ff01-46ae-a200-1332dae43706`::ref 0 aborting False >Thread-89::DEBUG::2014-01-22 16:49:04,103::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-89::DEBUG::2014-01-22 16:49:04,103::task::579::TaskManager.Task::(_updateState) Task=`e02cbbd3-8f44-48a6-b7fb-8555180bef19`::moving from state init -> state preparing >Thread-89::INFO::2014-01-22 16:49:04,104::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainsList(spUUID='00000000-0000-0000-0000-000000000000', domainClass=None, storageType=None, remotePath=None, options=None) >Thread-89::DEBUG::2014-01-22 16:49:04,104::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-89::DEBUG::2014-01-22 16:49:04,104::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-89::DEBUG::2014-01-22 16:49:04,104::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-89::DEBUG::2014-01-22 16:49:04,104::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-89::DEBUG::2014-01-22 16:49:04,105::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-89::DEBUG::2014-01-22 16:49:04,134::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-89::DEBUG::2014-01-22 16:49:04,135::misc::819::SamplingMethod::(__call__) Returning last result >Thread-89::DEBUG::2014-01-22 16:49:09,182::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-89::DEBUG::2014-01-22 16:49:09,301::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-89::DEBUG::2014-01-22 16:49:09,302::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-89::DEBUG::2014-01-22 16:49:09,308::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-89::DEBUG::2014-01-22 16:49:09,308::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-89::DEBUG::2014-01-22 16:49:09,309::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-89::DEBUG::2014-01-22 16:49:09,309::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-89::DEBUG::2014-01-22 16:49:09,309::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-89::DEBUG::2014-01-22 16:49:09,309::misc::819::SamplingMethod::(__call__) Returning last result >Thread-89::DEBUG::2014-01-22 16:49:09,310::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-89::DEBUG::2014-01-22 16:49:09,312::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-89::DEBUG::2014-01-22 16:49:09,421::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >Thread-89::DEBUG::2014-01-22 16:49:09,422::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-89::DEBUG::2014-01-22 16:49:09,431::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-89::DEBUG::2014-01-22 16:49:09,433::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-89::DEBUG::2014-01-22 16:49:09,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-89::DEBUG::2014-01-22 16:49:09,443::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-89::INFO::2014-01-22 16:49:09,443::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS already registered >Thread-89::INFO::2014-01-22 16:49:09,443::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS already registered >Thread-89::INFO::2014-01-22 16:49:09,444::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainsList, Return response: {'domlist': ['3900e952-713e-4ea9-b32c-097cd4994bc5']} >Thread-89::DEBUG::2014-01-22 16:49:09,444::task::1168::TaskManager.Task::(prepare) Task=`e02cbbd3-8f44-48a6-b7fb-8555180bef19`::finished: {'domlist': ['3900e952-713e-4ea9-b32c-097cd4994bc5']} >Thread-89::DEBUG::2014-01-22 16:49:09,444::task::579::TaskManager.Task::(_updateState) Task=`e02cbbd3-8f44-48a6-b7fb-8555180bef19`::moving from state preparing -> state finished >Thread-89::DEBUG::2014-01-22 16:49:09,444::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-89::DEBUG::2014-01-22 16:49:09,445::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-89::DEBUG::2014-01-22 16:49:09,445::task::974::TaskManager.Task::(_decref) Task=`e02cbbd3-8f44-48a6-b7fb-8555180bef19`::ref 0 aborting False >Thread-92::DEBUG::2014-01-22 16:49:09,510::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-92::DEBUG::2014-01-22 16:49:09,511::task::579::TaskManager.Task::(_updateState) Task=`9115fdee-2664-40e3-9aec-870e4fb55983`::moving from state init -> state preparing >Thread-92::INFO::2014-01-22 16:49:09,511::logUtils::44::dispatcher::(wrapper) Run and protect: getStorageDomainInfo(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', options=None) >Thread-92::INFO::2014-01-22 16:49:09,511::fileSD::299::Storage.StorageDomain::(validate) sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-92::DEBUG::2014-01-22 16:49:09,519::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-92::DEBUG::2014-01-22 16:49:09,520::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`5cf56578-6fcd-4395-ba3a-25a43bac97a1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2759' at 'getStorageDomainInfo' >Thread-92::DEBUG::2014-01-22 16:49:09,520::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'shared' >Thread-92::DEBUG::2014-01-22 16:49:09,521::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'shared' (1 active user) >Thread-92::DEBUG::2014-01-22 16:49:09,521::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`5cf56578-6fcd-4395-ba3a-25a43bac97a1`::Granted request >Thread-92::DEBUG::2014-01-22 16:49:09,522::task::811::TaskManager.Task::(resourceAcquired) Task=`9115fdee-2664-40e3-9aec-870e4fb55983`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (shared) >Thread-92::DEBUG::2014-01-22 16:49:09,522::task::974::TaskManager.Task::(_decref) Task=`9115fdee-2664-40e3-9aec-870e4fb55983`::ref 1 aborting False >Thread-92::INFO::2014-01-22 16:49:09,525::logUtils::47::dispatcher::(wrapper) Run and protect: getStorageDomainInfo, Return response: {'info': {'uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'master_ver': 0, 'lver': -1, 'version': '3', 'role': 'Master', 'remotePath': '10.34.63.199:/jbelka/hosted', 'spm_id': -1, 'type': 'NFS', 'class': 'Data', 'pool': ['e1263145-a6a9-4993-9d8a-cd92a3e5c1fb'], 'name': 'hosted_storage'}} >Thread-92::DEBUG::2014-01-22 16:49:09,525::task::1168::TaskManager.Task::(prepare) Task=`9115fdee-2664-40e3-9aec-870e4fb55983`::finished: {'info': {'uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'master_ver': 0, 'lver': -1, 'version': '3', 'role': 'Master', 'remotePath': '10.34.63.199:/jbelka/hosted', 'spm_id': -1, 'type': 'NFS', 'class': 'Data', 'pool': ['e1263145-a6a9-4993-9d8a-cd92a3e5c1fb'], 'name': 'hosted_storage'}} >Thread-92::DEBUG::2014-01-22 16:49:09,526::task::579::TaskManager.Task::(_updateState) Task=`9115fdee-2664-40e3-9aec-870e4fb55983`::moving from state preparing -> state finished >Thread-92::DEBUG::2014-01-22 16:49:09,526::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>} >Thread-92::DEBUG::2014-01-22 16:49:09,526::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-92::DEBUG::2014-01-22 16:49:09,527::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >Thread-92::DEBUG::2014-01-22 16:49:09,527::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >Thread-92::DEBUG::2014-01-22 16:49:09,527::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >Thread-92::DEBUG::2014-01-22 16:49:09,527::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >Thread-92::DEBUG::2014-01-22 16:49:09,528::task::974::TaskManager.Task::(_decref) Task=`9115fdee-2664-40e3-9aec-870e4fb55983`::ref 0 aborting False >Thread-93::DEBUG::2014-01-22 16:49:11,507::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-93::DEBUG::2014-01-22 16:49:11,508::task::579::TaskManager.Task::(_updateState) Task=`b7418950-2262-4acc-b165-1bbd7d311802`::moving from state init -> state preparing >Thread-93::INFO::2014-01-22 16:49:11,508::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', hostID=1, scsiKey='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', msdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', masterVersion=1, options=None) >Thread-93::DEBUG::2014-01-22 16:49:11,508::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`3a7f3f6b-7f1b-47de-b02f-ee04c03e0dda`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool' >Thread-93::DEBUG::2014-01-22 16:49:11,509::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-93::DEBUG::2014-01-22 16:49:11,509::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-93::DEBUG::2014-01-22 16:49:11,509::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`3a7f3f6b-7f1b-47de-b02f-ee04c03e0dda`::Granted request >Thread-93::DEBUG::2014-01-22 16:49:11,509::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`bf917c4f-9586-4e3a-bd1d-b7530bcae5d4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool' >Thread-93::DEBUG::2014-01-22 16:49:11,509::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-93::DEBUG::2014-01-22 16:49:11,509::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-93::DEBUG::2014-01-22 16:49:11,510::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`bf917c4f-9586-4e3a-bd1d-b7530bcae5d4`::Granted request >Thread-93::INFO::2014-01-22 16:49:11,510::sp::674::Storage.StoragePool::(connect) Connect host #1 to the storage pool e1263145-a6a9-4993-9d8a-cd92a3e5c1fb with master domain: 3900e952-713e-4ea9-b32c-097cd4994bc5 (ver = 1) >Thread-93::DEBUG::2014-01-22 16:49:11,510::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:11,510::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:11,510::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:11,510::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:11,511::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:11,511::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:11,511::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-93::DEBUG::2014-01-22 16:49:11,511::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-93::DEBUG::2014-01-22 16:49:11,511::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-93::DEBUG::2014-01-22 16:49:11,511::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-93::DEBUG::2014-01-22 16:49:11,511::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-93::DEBUG::2014-01-22 16:49:11,524::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-93::DEBUG::2014-01-22 16:49:11,524::misc::819::SamplingMethod::(__call__) Returning last result >Thread-93::DEBUG::2014-01-22 16:49:15,566::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-93::DEBUG::2014-01-22 16:49:15,648::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-93::DEBUG::2014-01-22 16:49:15,648::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:15,649::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:15,649::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:15,649::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:15,650::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:15,650::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:15,650::misc::819::SamplingMethod::(__call__) Returning last result >Thread-93::DEBUG::2014-01-22 16:49:15,656::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-93::DEBUG::2014-01-22 16:49:15,657::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-93::DEBUG::2014-01-22 16:49:15,665::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-93::DEBUG::2014-01-22 16:49:15,667::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-93::INFO::2014-01-22 16:49:15,667::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS already registered >Thread-93::INFO::2014-01-22 16:49:15,667::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS already registered >Thread-93::DEBUG::2014-01-22 16:49:15,668::sp::1529::Storage.StoragePool::(getMasterDomain) Master domain 3900e952-713e-4ea9-b32c-097cd4994bc5 verified, version 1 >Thread-93::DEBUG::2014-01-22 16:49:15,668::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-93::DEBUG::2014-01-22 16:49:15,668::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-93::DEBUG::2014-01-22 16:49:15,675::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-93::INFO::2014-01-22 16:49:15,676::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-93::DEBUG::2014-01-22 16:49:15,677::sp::1562::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `e1263145-a6a9-4993-9d8a-cd92a3e5c1fb` started monitoring domain `3900e952-713e-4ea9-b32c-097cd4994bc5` >Thread-95::DEBUG::2014-01-22 16:49:15,677::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-93::DEBUG::2014-01-22 16:49:15,678::misc::819::SamplingMethod::(__call__) Returning last result >Thread-93::DEBUG::2014-01-22 16:49:15,681::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-93::DEBUG::2014-01-22 16:49:15,691::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-95::DEBUG::2014-01-22 16:49:15,704::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-95::DEBUG::2014-01-22 16:49:15,710::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000296028 s, 1.6 MB/s\n'; <rc> = 0 >Thread-95::DEBUG::2014-01-22 16:49:15,714::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3900e952-713e-4ea9-b32c-097cd4994bc5 changed its status to Valid >Thread-96::DEBUG::2014-01-22 16:49:15,714::misc::931::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-95::INFO::2014-01-22 16:49:15,714::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-96::DEBUG::2014-01-22 16:49:15,715::misc::941::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-95::DEBUG::2014-01-22 16:49:15,715::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >Thread-96::DEBUG::2014-01-22 16:49:15,716::misc::951::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-97::INFO::2014-01-22 16:49:15,721::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-93::DEBUG::2014-01-22 16:49:15,783::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >Thread-93::DEBUG::2014-01-22 16:49:15,783::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-93::INFO::2014-01-22 16:49:15,786::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-93::INFO::2014-01-22 16:49:15,786::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd >Thread-93::DEBUG::2014-01-22 16:49:15,787::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-93::DEBUG::2014-01-22 16:49:15,787::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-93::DEBUG::2014-01-22 16:49:15,787::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-93::DEBUG::2014-01-22 16:49:15,788::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-93::DEBUG::2014-01-22 16:49:15,788::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-93::DEBUG::2014-01-22 16:49:15,788::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-93::DEBUG::2014-01-22 16:49:15,789::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-93::DEBUG::2014-01-22 16:49:15,789::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-93::INFO::2014-01-22 16:49:15,789::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-93::DEBUG::2014-01-22 16:49:15,789::task::1168::TaskManager.Task::(prepare) Task=`b7418950-2262-4acc-b165-1bbd7d311802`::finished: True >Thread-93::DEBUG::2014-01-22 16:49:15,790::task::579::TaskManager.Task::(_updateState) Task=`b7418950-2262-4acc-b165-1bbd7d311802`::moving from state preparing -> state finished >Thread-93::DEBUG::2014-01-22 16:49:15,790::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-93::DEBUG::2014-01-22 16:49:15,790::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-93::DEBUG::2014-01-22 16:49:15,790::task::974::TaskManager.Task::(_decref) Task=`b7418950-2262-4acc-b165-1bbd7d311802`::ref 0 aborting False >Thread-99::DEBUG::2014-01-22 16:49:15,843::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-99::DEBUG::2014-01-22 16:49:15,843::task::579::TaskManager.Task::(_updateState) Task=`ba070901-6abd-4e69-bc97-a21847bd8103`::moving from state init -> state preparing >Thread-99::INFO::2014-01-22 16:49:15,844::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-99::DEBUG::2014-01-22 16:49:15,844::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`c375ed2c-33b4-49f2-ba61-6df7d0986bf7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2562' at 'getStoragePoolInfo' >Thread-99::DEBUG::2014-01-22 16:49:15,845::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'shared' >Thread-99::DEBUG::2014-01-22 16:49:15,845::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'shared' (1 active user) >Thread-99::DEBUG::2014-01-22 16:49:15,845::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`c375ed2c-33b4-49f2-ba61-6df7d0986bf7`::Granted request >Thread-99::DEBUG::2014-01-22 16:49:15,846::task::811::TaskManager.Task::(resourceAcquired) Task=`ba070901-6abd-4e69-bc97-a21847bd8103`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (shared) >Thread-99::DEBUG::2014-01-22 16:49:15,846::task::974::TaskManager.Task::(_decref) Task=`ba070901-6abd-4e69-bc97-a21847bd8103`::ref 1 aborting False >Thread-99::INFO::2014-01-22 16:49:15,849::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': -1, 'master_uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'name': 'hosted_datacenter', 'version': '3', 'domains': '3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 0}, 'dominfo': {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'status': 'Active', 'diskfree': '75295358976', 'isoprefix': '', 'alerts': [], 'disktotal': '422943129600', 'version': 3}}} >Thread-99::DEBUG::2014-01-22 16:49:15,849::task::1168::TaskManager.Task::(prepare) Task=`ba070901-6abd-4e69-bc97-a21847bd8103`::finished: {'info': {'spm_id': -1, 'master_uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'name': 'hosted_datacenter', 'version': '3', 'domains': '3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 0}, 'dominfo': {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'status': 'Active', 'diskfree': '75295358976', 'isoprefix': '', 'alerts': [], 'disktotal': '422943129600', 'version': 3}}} >Thread-99::DEBUG::2014-01-22 16:49:15,849::task::579::TaskManager.Task::(_updateState) Task=`ba070901-6abd-4e69-bc97-a21847bd8103`::moving from state preparing -> state finished >Thread-99::DEBUG::2014-01-22 16:49:15,849::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-99::DEBUG::2014-01-22 16:49:15,849::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-99::DEBUG::2014-01-22 16:49:15,849::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-99::DEBUG::2014-01-22 16:49:15,850::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-99::DEBUG::2014-01-22 16:49:15,850::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-99::DEBUG::2014-01-22 16:49:15,850::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-99::DEBUG::2014-01-22 16:49:15,850::task::974::TaskManager.Task::(_decref) Task=`ba070901-6abd-4e69-bc97-a21847bd8103`::ref 0 aborting False >Thread-95::DEBUG::2014-01-22 16:49:25,726::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-95::DEBUG::2014-01-22 16:49:25,733::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000392087 s, 1.2 MB/s\n'; <rc> = 0 >Thread-95::DEBUG::2014-01-22 16:49:35,747::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-95::DEBUG::2014-01-22 16:49:35,755::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000323559 s, 1.5 MB/s\n'; <rc> = 0 >Thread-95::DEBUG::2014-01-22 16:49:45,768::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-95::DEBUG::2014-01-22 16:49:45,775::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000370232 s, 1.3 MB/s\n'; <rc> = 0 >Thread-95::DEBUG::2014-01-22 16:49:55,788::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-95::DEBUG::2014-01-22 16:49:55,795::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.00041593 s, 1.2 MB/s\n'; <rc> = 0 >MainThread::DEBUG::2014-01-22 16:49:56,948::vdsm::45::vds::(sigtermHandler) Received signal 15 >MainThread::INFO::2014-01-22 16:49:57,849::vmChannels::183::vds::(stop) VM channels listener was stopped. >MainThread::INFO::2014-01-22 16:49:57,850::momIF::84::MOM::(stop) Shutting down MOM >MainThread::DEBUG::2014-01-22 16:49:57,851::task::579::TaskManager.Task::(_updateState) Task=`66b0b892-1f89-4681-a36e-8b2994faee4f`::moving from state init -> state preparing >MainThread::INFO::2014-01-22 16:49:57,851::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) >Thread-11::DEBUG::2014-01-22 16:49:57,852::storageServer::698::ConnectionMonitor::(_monitorConnections) Monitoring stopped >MainThread::INFO::2014-01-22 16:49:58,433::domainMonitor::125::Storage.DomainMonitor::(close) Stopping domain monitors >MainThread::INFO::2014-01-22 16:49:58,433::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-95::DEBUG::2014-01-22 16:49:58,434::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-95::INFO::2014-01-22 16:49:58,434::clusterlock::197::SANLock::(releaseHostId) Releasing host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >VM Channels Listener::INFO::2014-01-22 16:49:58,830::vmChannels::178::vds::(run) VM channels listener thread has ended. >MainThread::INFO::2014-01-22 16:50:00,772::vdsm::101::vds::(run) (PID: 6106) I am the actual vdsm 4.13.2-0.6.el6ev dell-r210ii-13.rhev.lab.eng.brq.redhat.com (2.6.32-431.el6.x86_64) >MainThread::DEBUG::2014-01-22 16:50:00,939::resourceManager::420::ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2014-01-22 16:50:00,939::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 >MainThread::WARNING::2014-01-22 16:50:00,942::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists >MainThread::DEBUG::2014-01-22 16:50:01,255::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >MainThread::DEBUG::2014-01-22 16:50:01,272::multipath::144::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving >MainThread::DEBUG::2014-01-22 16:50:01,272::hsm::427::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2014-01-22 16:50:01,392::hsm::427::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-01-22 16:50:01,392::hsm::451::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >MainThread::DEBUG::2014-01-22 16:50:01,396::hsm::483::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >MainThread::DEBUG::2014-01-22 16:50:01,396::hsm::484::Storage.HSM::(__cleanStorageRepository) Mount list: ['/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted'] >MainThread::DEBUG::2014-01-22 16:50:01,396::hsm::486::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >MainThread::DEBUG::2014-01-22 16:50:01,399::hsm::529::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >storageRefresh::DEBUG::2014-01-22 16:50:01,400::lvm::338::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >MainThread::INFO::2014-01-22 16:50:01,401::dispatcher::98::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >MainThread::DEBUG::2014-01-22 16:50:01,403::task::579::TaskManager.Task::(_updateState) Task=`28dd9a26-8ddf-41fd-a5f3-c80cb68ac468`::moving from state init -> state preparing >MainThread::INFO::2014-01-22 16:50:01,403::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x7f7aa47674d0>>) >MainThread::INFO::2014-01-22 16:50:01,404::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None >MainThread::DEBUG::2014-01-22 16:50:01,404::task::1168::TaskManager.Task::(prepare) Task=`28dd9a26-8ddf-41fd-a5f3-c80cb68ac468`::finished: None >MainThread::DEBUG::2014-01-22 16:50:01,404::task::579::TaskManager.Task::(_updateState) Task=`28dd9a26-8ddf-41fd-a5f3-c80cb68ac468`::moving from state preparing -> state finished >MainThread::DEBUG::2014-01-22 16:50:01,404::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2014-01-22 16:50:01,404::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2014-01-22 16:50:01,405::task::974::TaskManager.Task::(_decref) Task=`28dd9a26-8ddf-41fd-a5f3-c80cb68ac468`::ref 0 aborting False >storageRefresh::DEBUG::2014-01-22 16:50:01,405::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >MainThread::INFO::2014-01-22 16:50:01,416::momIF::47::MOM::(__init__) Starting up MOM >MainThread::INFO::2014-01-22 16:50:01,419::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds. >clientIFinit::DEBUG::2014-01-22 16:50:01,420::libvirtconnection::131::libvirtconnection::(get) trying to connect libvirt >VM Channels Listener::INFO::2014-01-22 16:50:01,427::vmChannels::170::vds::(run) Starting VM channels listener thread. >clientIFinit::DEBUG::2014-01-22 16:50:01,459::clientIF::554::vds::(_cleanOldFiles) removing old file 277d29a5-5802-4e2f-9852-ebb6e785262f.recovery >storageRefresh::DEBUG::2014-01-22 16:50:01,512::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:50:01,513::lvm::363::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:01,515::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:01,517::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:50:01,602::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:50:01,605::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:01,606::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:50:01,690::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:50:01,691::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:01,692::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:50:01,787::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:50:01,787::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:01,788::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >storageRefresh::DEBUG::2014-01-22 16:50:01,788::misc::811::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-01-22 16:50:01,788::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >storageRefresh::DEBUG::2014-01-22 16:50:01,788::misc::811::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-01-22 16:50:01,789::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:50:01,803::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >storageRefresh::DEBUG::2014-01-22 16:50:01,803::misc::819::SamplingMethod::(__call__) Returning last result >Thread-13::DEBUG::2014-01-22 16:50:01,812::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call getHardwareInfo with () {} >Thread-13::DEBUG::2014-01-22 16:50:01,817::BindingXMLRPC::991::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210 II', 'systemSerialNumber': '34SRJ5J', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0034-5310-8052-B3C04F4A354A', 'systemManufacturer': 'Dell Inc.'}} >Thread-14::DEBUG::2014-01-22 16:50:01,871::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-14::DEBUG::2014-01-22 16:50:01,873::task::579::TaskManager.Task::(_updateState) Task=`ada12bf1-a209-46d8-9357-840da5188fd7`::moving from state init -> state preparing >Thread-14::INFO::2014-01-22 16:50:01,873::logUtils::44::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >Thread-14::INFO::2014-01-22 16:50:01,873::logUtils::47::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >Thread-14::DEBUG::2014-01-22 16:50:01,873::task::1168::TaskManager.Task::(prepare) Task=`ada12bf1-a209-46d8-9357-840da5188fd7`::finished: {'poollist': []} >Thread-14::DEBUG::2014-01-22 16:50:01,874::task::579::TaskManager.Task::(_updateState) Task=`ada12bf1-a209-46d8-9357-840da5188fd7`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-01-22 16:50:01,874::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-01-22 16:50:01,874::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-01-22 16:50:01,874::task::974::TaskManager.Task::(_decref) Task=`ada12bf1-a209-46d8-9357-840da5188fd7`::ref 0 aborting False >Thread-15::DEBUG::2014-01-22 16:50:01,930::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-15::DEBUG::2014-01-22 16:50:01,930::task::579::TaskManager.Task::(_updateState) Task=`e6ad0ca4-1a49-4024-97f9-792aadc9f65e`::moving from state init -> state preparing >Thread-15::INFO::2014-01-22 16:50:01,931::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '', 'connection': '10.34.63.199:/jbelka/hosted', 'iqn': '', 'portal': '', 'user': 'kvm', 'protocol_version': '3', 'password': '******', 'id': '9e10ccab-8a15-48c2-b1cc-49511bf53a28'}], options=None) >Thread-15::DEBUG::2014-01-22 16:50:01,936::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted >Thread-15::DEBUG::2014-01-22 16:50:01,992::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('3900e952-713e-4ea9-b32c-097cd4994bc5',) >Thread-15::DEBUG::2014-01-22 16:50:01,992::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {3900e952-713e-4ea9-b32c-097cd4994bc5: storage.nfsSD.findDomain} >Thread-15::INFO::2014-01-22 16:50:01,992::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '9e10ccab-8a15-48c2-b1cc-49511bf53a28'}]} >Thread-15::DEBUG::2014-01-22 16:50:01,993::task::1168::TaskManager.Task::(prepare) Task=`e6ad0ca4-1a49-4024-97f9-792aadc9f65e`::finished: {'statuslist': [{'status': 0, 'id': '9e10ccab-8a15-48c2-b1cc-49511bf53a28'}]} >Thread-15::DEBUG::2014-01-22 16:50:01,993::task::579::TaskManager.Task::(_updateState) Task=`e6ad0ca4-1a49-4024-97f9-792aadc9f65e`::moving from state preparing -> state finished >Thread-15::DEBUG::2014-01-22 16:50:01,993::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15::DEBUG::2014-01-22 16:50:01,993::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15::DEBUG::2014-01-22 16:50:01,994::task::974::TaskManager.Task::(_decref) Task=`e6ad0ca4-1a49-4024-97f9-792aadc9f65e`::ref 0 aborting False >Thread-16::DEBUG::2014-01-22 16:50:02,051::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-16::DEBUG::2014-01-22 16:50:02,052::task::579::TaskManager.Task::(_updateState) Task=`1746f050-0a3d-45c9-b627-afbca575dbc7`::moving from state init -> state preparing >Thread-16::INFO::2014-01-22 16:50:02,052::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', hostID=1, scsiKey='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', msdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', masterVersion=1, options=None) >Thread-16::DEBUG::2014-01-22 16:50:02,053::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`2c50f9a1-7411-4fff-96c3-7abd8d06b86a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool' >Thread-16::DEBUG::2014-01-22 16:50:02,053::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-16::DEBUG::2014-01-22 16:50:02,053::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-16::DEBUG::2014-01-22 16:50:02,053::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`2c50f9a1-7411-4fff-96c3-7abd8d06b86a`::Granted request >Thread-16::DEBUG::2014-01-22 16:50:02,054::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`28483be1-85c9-4ca8-83b9-0b0142ded7c8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool' >Thread-16::DEBUG::2014-01-22 16:50:02,055::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-16::DEBUG::2014-01-22 16:50:02,055::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-16::DEBUG::2014-01-22 16:50:02,055::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`28483be1-85c9-4ca8-83b9-0b0142ded7c8`::Granted request >Thread-16::INFO::2014-01-22 16:50:02,056::sp::674::Storage.StoragePool::(connect) Connect host #1 to the storage pool e1263145-a6a9-4993-9d8a-cd92a3e5c1fb with master domain: 3900e952-713e-4ea9-b32c-097cd4994bc5 (ver = 1) >Thread-16::DEBUG::2014-01-22 16:50:02,056::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:02,057::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:02,057::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:02,057::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:02,057::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:02,058::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:02,058::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >storageRefresh::DEBUG::2014-01-22 16:50:05,855::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:50:05,981::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-01-22 16:50:05,982::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:05,982::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:05,983::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:05,983::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:05,983::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:05,984::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-01-22 16:50:05,984::misc::819::SamplingMethod::(__call__) Returning last result >Thread-16::DEBUG::2014-01-22 16:50:05,984::misc::811::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::WARNING::2014-01-22 16:50:05,986::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists >Thread-16::DEBUG::2014-01-22 16:50:05,986::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-16::DEBUG::2014-01-22 16:50:05,990::misc::811::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-01-22 16:50:05,992::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`8b503aad-ce56-44ce-b202-2810e62cde34`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool' >storageRefresh::DEBUG::2014-01-22 16:50:05,994::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-16::DEBUG::2014-01-22 16:50:05,994::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >storageRefresh::DEBUG::2014-01-22 16:50:06,002::resourceManager::576::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is currently locked, Entering queue (1 in queue) >Thread-16::DEBUG::2014-01-22 16:50:06,009::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 >Thread-16::DEBUG::2014-01-22 16:50:06,009::misc::819::SamplingMethod::(__call__) Returning last result >Thread-16::DEBUG::2014-01-22 16:50:11,050::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-16::DEBUG::2014-01-22 16:50:11,136::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-16::DEBUG::2014-01-22 16:50:11,137::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:11,138::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:11,138::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:11,138::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:11,139::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:11,139::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:11,139::misc::819::SamplingMethod::(__call__) Returning last result >Thread-16::DEBUG::2014-01-22 16:50:11,201::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-16::DEBUG::2014-01-22 16:50:11,252::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-16::DEBUG::2014-01-22 16:50:11,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-16::DEBUG::2014-01-22 16:50:11,264::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-16::DEBUG::2014-01-22 16:50:11,265::resourceManager::420::ResourceManager::(registerNamespace) Registering namespace '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS' >Thread-16::DEBUG::2014-01-22 16:50:11,265::resourceManager::420::ResourceManager::(registerNamespace) Registering namespace '3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS' >Thread-16::DEBUG::2014-01-22 16:50:11,266::sp::1529::Storage.StoragePool::(getMasterDomain) Master domain 3900e952-713e-4ea9-b32c-097cd4994bc5 verified, version 1 >Thread-16::DEBUG::2014-01-22 16:50:11,266::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-16::DEBUG::2014-01-22 16:50:11,266::misc::811::SamplingMethod::(__call__) Got in to sampling method >Thread-16::DEBUG::2014-01-22 16:50:11,272::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >Thread-16::INFO::2014-01-22 16:50:11,272::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-16::DEBUG::2014-01-22 16:50:11,273::sp::1562::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `e1263145-a6a9-4993-9d8a-cd92a3e5c1fb` started monitoring domain `3900e952-713e-4ea9-b32c-097cd4994bc5` >Thread-18::DEBUG::2014-01-22 16:50:11,273::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-16::DEBUG::2014-01-22 16:50:11,274::misc::819::SamplingMethod::(__call__) Returning last result >Thread-16::WARNING::2014-01-22 16:50:11,275::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb already exists >Thread-16::DEBUG::2014-01-22 16:50:11,284::lvm::386::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-16::DEBUG::2014-01-22 16:50:11,292::lvm::309::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >Thread-18::DEBUG::2014-01-22 16:50:11,301::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-18::DEBUG::2014-01-22 16:50:11,306::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000277903 s, 1.7 MB/s\n'; <rc> = 0 >Thread-18::DEBUG::2014-01-22 16:50:11,310::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3900e952-713e-4ea9-b32c-097cd4994bc5 changed its status to Valid >Thread-19::DEBUG::2014-01-22 16:50:11,311::misc::931::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-18::INFO::2014-01-22 16:50:11,311::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-19::DEBUG::2014-01-22 16:50:11,312::misc::941::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-19::DEBUG::2014-01-22 16:50:11,312::misc::951::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-18::DEBUG::2014-01-22 16:50:11,312::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >Thread-20::INFO::2014-01-22 16:50:11,314::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-16::DEBUG::2014-01-22 16:50:11,393::lvm::309::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >Thread-16::DEBUG::2014-01-22 16:50:11,394::lvm::428::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16::INFO::2014-01-22 16:50:11,396::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-16::INFO::2014-01-22 16:50:11,396::sp::1192::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 to /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd >Thread-16::DEBUG::2014-01-22 16:50:11,397::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-16::DEBUG::2014-01-22 16:50:11,397::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-16::DEBUG::2014-01-22 16:50:11,397::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-16::DEBUG::2014-01-22 16:50:11,397::resourceManager::653::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' has 1 requests in queue. Handling top request. >Thread-16::DEBUG::2014-01-22 16:50:11,397::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`8b503aad-ce56-44ce-b202-2810e62cde34`::Granted request >Thread-16::DEBUG::2014-01-22 16:50:11,398::resourceManager::685::ResourceManager::(releaseResource) Request 'Request for Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb - exclusive: granted' was granted >Thread-16::DEBUG::2014-01-22 16:50:11,400::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >storageRefresh::DEBUG::2014-01-22 16:50:11,401::sp::1529::Storage.StoragePool::(getMasterDomain) Master domain 3900e952-713e-4ea9-b32c-097cd4994bc5 verified, version 1 >Thread-16::DEBUG::2014-01-22 16:50:11,403::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >storageRefresh::DEBUG::2014-01-22 16:50:11,403::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-16::DEBUG::2014-01-22 16:50:11,404::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-16::DEBUG::2014-01-22 16:50:11,405::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >storageRefresh::DEBUG::2014-01-22 16:50:11,405::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-16::INFO::2014-01-22 16:50:11,406::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >storageRefresh::DEBUG::2014-01-22 16:50:11,407::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-16::DEBUG::2014-01-22 16:50:11,408::task::1168::TaskManager.Task::(prepare) Task=`1746f050-0a3d-45c9-b627-afbca575dbc7`::finished: True >storageRefresh::DEBUG::2014-01-22 16:50:11,408::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-16::DEBUG::2014-01-22 16:50:11,409::task::579::TaskManager.Task::(_updateState) Task=`1746f050-0a3d-45c9-b627-afbca575dbc7`::moving from state preparing -> state finished >Thread-16::DEBUG::2014-01-22 16:50:11,410::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-16::DEBUG::2014-01-22 16:50:11,410::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-16::DEBUG::2014-01-22 16:50:11,410::task::974::TaskManager.Task::(_decref) Task=`1746f050-0a3d-45c9-b627-afbca575dbc7`::ref 0 aborting False >Thread-22::DEBUG::2014-01-22 16:50:11,460::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-22::DEBUG::2014-01-22 16:50:11,461::task::579::TaskManager.Task::(_updateState) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::moving from state init -> state preparing >Thread-22::INFO::2014-01-22 16:50:11,461::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', prevID=-1, prevLVER=-1, recoveryMode=None, scsiFencing='false', maxHostID=250, domVersion=3, options=None) >Thread-22::DEBUG::2014-01-22 16:50:11,462::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`7031ba35-afaf-43cd-ae79-24739bde72b9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '587' at 'spmStart' >Thread-22::DEBUG::2014-01-22 16:50:11,462::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-22::DEBUG::2014-01-22 16:50:11,462::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-22::DEBUG::2014-01-22 16:50:11,462::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`7031ba35-afaf-43cd-ae79-24739bde72b9`::Granted request >Thread-22::DEBUG::2014-01-22 16:50:11,463::task::811::TaskManager.Task::(resourceAcquired) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-22::DEBUG::2014-01-22 16:50:11,463::task::974::TaskManager.Task::(_decref) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::ref 1 aborting False >Thread-22::INFO::2014-01-22 16:50:11,463::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-22::DEBUG::2014-01-22 16:50:11,464::task::1163::TaskManager.Task::(prepare) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::Prepare: 1 jobs exist, move to acquiring >Thread-22::DEBUG::2014-01-22 16:50:11,464::task::579::TaskManager.Task::(_updateState) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::moving from state preparing -> state acquiring >Thread-22::DEBUG::2014-01-22 16:50:11,464::task::579::TaskManager.Task::(_updateState) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::moving from state acquiring -> state queued >Thread-22::DEBUG::2014-01-22 16:50:11,464::taskManager::50::TaskManager::(_queueTask) queuing task: a783aea1-c48b-4931-8808-7a5351679c35 >Thread-22::DEBUG::2014-01-22 16:50:11,465::taskManager::56::TaskManager::(_queueTask) task queued: a783aea1-c48b-4931-8808-7a5351679c35 >Thread-1::DEBUG::2014-01-22 16:50:11,465::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-22::DEBUG::2014-01-22 16:50:11,466::task::1165::TaskManager.Task::(prepare) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::returning >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,467::threadPool::205::Misc.ThreadPool.WorkerThread::(run) Task: a783aea1-c48b-4931-8808-7a5351679c35 running: <bound method Task.commit of <storage.task.Task instance at 0x22d5128>> with: None >Thread-22::DEBUG::2014-01-22 16:50:11,468::task::974::TaskManager.Task::(_decref) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::ref 0 aborting False >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,469::task::1176::TaskManager.Task::(commit) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::committing task: a783aea1-c48b-4931-8808-7a5351679c35 >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,471::task::579::TaskManager.Task::(_updateState) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::moving from state queued -> state running >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,473::task::889::TaskManager.Task::(_runJobs) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f7a500bbed0>> (args: (-1, -1, 'false', 250, 3) kwargs: {}) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,475::task::315::TaskManager.Task::(run) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f7a500bbed0>> (args: (-1, -1, 'false', 250, 3) kwargs: {}) callback None >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,476::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,477::misc::811::SamplingMethod::(__call__) Got in to sampling method >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,485::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,486::misc::819::SamplingMethod::(__call__) Returning last result >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:11,496::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >a783aea1-c48b-4931-8808-7a5351679c35::INFO::2014-01-22 16:50:11,497::sp::260::Storage.StoragePool::(startSpm) expected previd:-1 lver:0 got request for previd:-1 lver:-1 >a783aea1-c48b-4931-8808-7a5351679c35::INFO::2014-01-22 16:50:11,497::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-23::DEBUG::2014-01-22 16:50:11,527::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-23::DEBUG::2014-01-22 16:50:11,527::task::579::TaskManager.Task::(_updateState) Task=`0e2ac133-9a4c-4198-b688-fb9db30a66e8`::moving from state init -> state preparing >Thread-23::INFO::2014-01-22 16:50:11,528::logUtils::44::dispatcher::(wrapper) Run and protect: activateStorageDomain(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-23::DEBUG::2014-01-22 16:50:11,528::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`edd84083-4c61-4e54-a381-6bd86188cc29`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1239' at 'activateStorageDomain' >Thread-23::DEBUG::2014-01-22 16:50:11,528::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-23::DEBUG::2014-01-22 16:50:11,528::resourceManager::576::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is currently locked, Entering queue (1 in queue) >Thread-18::DEBUG::2014-01-22 16:50:21,323::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-18::DEBUG::2014-01-22 16:50:21,330::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000345585 s, 1.4 MB/s\n'; <rc> = 0 >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,499::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >a783aea1-c48b-4931-8808-7a5351679c35::INFO::2014-01-22 16:50:32,501::clusterlock::225::SANLock::(acquire) Acquiring cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,536::clusterlock::251::SANLock::(acquire) Cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,536::sp::272::Storage.StoragePool::(startSpm) spm lock acquired successfully >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,544::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bd730a1a9e954c62cc11ff6647668de4b92267cf'] >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,544::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,545::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,545::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=32677e398521f3f8dd8dc889ecc6de7d9a68f176'] >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,556::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,557::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`dcc61347-26a5-49c2-a4d4-6a6e5587b8e2`::Request was made in '/usr/share/vdsm/storage/sp.py' line '432' at '_upgradePool' >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,557::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,558::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,558::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`dcc61347-26a5-49c2-a4d4-6a6e5587b8e2`::Granted request >a783aea1-c48b-4931-8808-7a5351679c35::INFO::2014-01-22 16:50:32,558::sp::438::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `3900e952-713e-4ea9-b32c-097cd4994bc5` >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,559::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`65466f17-645f-4f5e-8003-3c991787bd24`::Request was made in '/usr/share/vdsm/storage/sp.py' line '440' at '_upgradePool' >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,559::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'exclusive' >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,559::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'exclusive' (1 active user) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,560::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`65466f17-645f-4f5e-8003-3c991787bd24`::Granted request >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,566::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,567::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,574::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=32677e398521f3f8dd8dc889ecc6de7d9a68f176'] >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,576::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >a783aea1-c48b-4931-8808-7a5351679c35::INFO::2014-01-22 16:50:32,577::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS already registered >a783aea1-c48b-4931-8808-7a5351679c35::INFO::2014-01-22 16:50:32,577::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS already registered >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,577::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,578::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,578::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,578::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,578::sp::443::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,579::sp::450::Storage.StoragePool::(_upgradePool) Registering with state change event >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,579::sp::453::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,579::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,579::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,579::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,580::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,580::sd::620::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/master/vms >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,581::sd::623::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/master/tasks >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,583::sp::333::Storage.StoragePool::(startSpm) ended. >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,584::task::579::TaskManager.Task::(_updateState) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::moving from state running -> state finished >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,584::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,584::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,584::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,585::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,585::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,585::resourceManager::653::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' has 1 requests in queue. Handling top request. >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,585::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`edd84083-4c61-4e54-a381-6bd86188cc29`::Granted request >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,586::resourceManager::685::ResourceManager::(releaseResource) Request 'Request for Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb - exclusive: granted' was granted >Thread-23::DEBUG::2014-01-22 16:50:32,587::task::811::TaskManager.Task::(resourceAcquired) Task=`0e2ac133-9a4c-4198-b688-fb9db30a66e8`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,587::task::897::TaskManager.Task::(_runJobs) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::Task.run: exit - success: result >Thread-23::DEBUG::2014-01-22 16:50:32,588::task::974::TaskManager.Task::(_decref) Task=`0e2ac133-9a4c-4198-b688-fb9db30a66e8`::ref 1 aborting False >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,589::task::974::TaskManager.Task::(_decref) Task=`a783aea1-c48b-4931-8808-7a5351679c35`::ref 0 aborting False >a783aea1-c48b-4931-8808-7a5351679c35::DEBUG::2014-01-22 16:50:32,590::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-23::DEBUG::2014-01-22 16:50:32,592::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`3f6ad3df-1c97-4fab-9381-f52f98b09f60`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1240' at 'activateStorageDomain' >Thread-23::DEBUG::2014-01-22 16:50:32,592::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'exclusive' >Thread-23::DEBUG::2014-01-22 16:50:32,592::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'exclusive' (1 active user) >Thread-23::DEBUG::2014-01-22 16:50:32,592::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`3f6ad3df-1c97-4fab-9381-f52f98b09f60`::Granted request >Thread-23::DEBUG::2014-01-22 16:50:32,593::task::811::TaskManager.Task::(resourceAcquired) Task=`0e2ac133-9a4c-4198-b688-fb9db30a66e8`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (exclusive) >Thread-23::DEBUG::2014-01-22 16:50:32,593::task::974::TaskManager.Task::(_decref) Task=`0e2ac133-9a4c-4198-b688-fb9db30a66e8`::ref 1 aborting False >Thread-23::INFO::2014-01-22 16:50:32,593::sp::1106::Storage.StoragePool::(activateSD) sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5 spUUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb >Thread-23::INFO::2014-01-22 16:50:32,594::logUtils::47::dispatcher::(wrapper) Run and protect: activateStorageDomain, Return response: None >Thread-23::DEBUG::2014-01-22 16:50:32,594::task::1168::TaskManager.Task::(prepare) Task=`0e2ac133-9a4c-4198-b688-fb9db30a66e8`::finished: None >Thread-23::DEBUG::2014-01-22 16:50:32,594::task::579::TaskManager.Task::(_updateState) Task=`0e2ac133-9a4c-4198-b688-fb9db30a66e8`::moving from state preparing -> state finished >Thread-23::DEBUG::2014-01-22 16:50:32,594::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>, 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-23::DEBUG::2014-01-22 16:50:32,595::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-23::DEBUG::2014-01-22 16:50:32,595::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >Thread-23::DEBUG::2014-01-22 16:50:32,595::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >Thread-23::DEBUG::2014-01-22 16:50:32,596::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >Thread-23::DEBUG::2014-01-22 16:50:32,596::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >Thread-23::DEBUG::2014-01-22 16:50:32,596::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-23::DEBUG::2014-01-22 16:50:32,596::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-23::DEBUG::2014-01-22 16:50:32,597::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-23::DEBUG::2014-01-22 16:50:32,597::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-23::DEBUG::2014-01-22 16:50:32,597::task::974::TaskManager.Task::(_decref) Task=`0e2ac133-9a4c-4198-b688-fb9db30a66e8`::ref 0 aborting False >Thread-25::DEBUG::2014-01-22 16:50:32,649::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-25::DEBUG::2014-01-22 16:50:32,649::task::579::TaskManager.Task::(_updateState) Task=`b7b53f7e-0de3-472e-b959-a9c92a92f2c8`::moving from state init -> state preparing >Thread-25::INFO::2014-01-22 16:50:32,650::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-25::DEBUG::2014-01-22 16:50:32,650::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-25::DEBUG::2014-01-22 16:50:32,650::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-25::INFO::2014-01-22 16:50:32,650::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-25::DEBUG::2014-01-22 16:50:32,651::task::1168::TaskManager.Task::(prepare) Task=`b7b53f7e-0de3-472e-b959-a9c92a92f2c8`::finished: {'allTasksStatus': {}} >Thread-25::DEBUG::2014-01-22 16:50:32,651::task::579::TaskManager.Task::(_updateState) Task=`b7b53f7e-0de3-472e-b959-a9c92a92f2c8`::moving from state preparing -> state finished >Thread-25::DEBUG::2014-01-22 16:50:32,651::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25::DEBUG::2014-01-22 16:50:32,651::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25::DEBUG::2014-01-22 16:50:32,651::task::974::TaskManager.Task::(_decref) Task=`b7b53f7e-0de3-472e-b959-a9c92a92f2c8`::ref 0 aborting False >Thread-26::DEBUG::2014-01-22 16:50:32,702::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-26::DEBUG::2014-01-22 16:50:32,702::task::579::TaskManager.Task::(_updateState) Task=`c6783d36-41da-4902-9292-d76a34d64070`::moving from state init -> state preparing >Thread-26::INFO::2014-01-22 16:50:32,703::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-26::INFO::2014-01-22 16:50:32,703::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}} >Thread-26::DEBUG::2014-01-22 16:50:32,703::task::1168::TaskManager.Task::(prepare) Task=`c6783d36-41da-4902-9292-d76a34d64070`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 1}} >Thread-26::DEBUG::2014-01-22 16:50:32,704::task::579::TaskManager.Task::(_updateState) Task=`c6783d36-41da-4902-9292-d76a34d64070`::moving from state preparing -> state finished >Thread-26::DEBUG::2014-01-22 16:50:32,704::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26::DEBUG::2014-01-22 16:50:32,704::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26::DEBUG::2014-01-22 16:50:32,704::task::974::TaskManager.Task::(_decref) Task=`c6783d36-41da-4902-9292-d76a34d64070`::ref 0 aborting False >Thread-27::DEBUG::2014-01-22 16:50:32,755::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-27::DEBUG::2014-01-22 16:50:32,755::task::579::TaskManager.Task::(_updateState) Task=`6ef8fc04-c8dc-4963-99ad-e2f4cc98166f`::moving from state init -> state preparing >Thread-27::INFO::2014-01-22 16:50:32,756::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-27::DEBUG::2014-01-22 16:50:32,756::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`1625c507-996f-4cfd-8d3f-ead26e83293b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2562' at 'getStoragePoolInfo' >Thread-27::DEBUG::2014-01-22 16:50:32,757::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'shared' >Thread-27::DEBUG::2014-01-22 16:50:32,757::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'shared' (1 active user) >Thread-27::DEBUG::2014-01-22 16:50:32,757::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`1625c507-996f-4cfd-8d3f-ead26e83293b`::Granted request >Thread-27::DEBUG::2014-01-22 16:50:32,757::task::811::TaskManager.Task::(resourceAcquired) Task=`6ef8fc04-c8dc-4963-99ad-e2f4cc98166f`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (shared) >Thread-27::DEBUG::2014-01-22 16:50:32,758::task::974::TaskManager.Task::(_decref) Task=`6ef8fc04-c8dc-4963-99ad-e2f4cc98166f`::ref 1 aborting False >Thread-27::INFO::2014-01-22 16:50:32,759::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'name': 'hosted_datacenter', 'version': '3', 'domains': '3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'status': 'Active', 'diskfree': '75295293440', 'isoprefix': '', 'alerts': [], 'disktotal': '422943129600', 'version': 3}}} >Thread-27::DEBUG::2014-01-22 16:50:32,760::task::1168::TaskManager.Task::(prepare) Task=`6ef8fc04-c8dc-4963-99ad-e2f4cc98166f`::finished: {'info': {'spm_id': 1, 'master_uuid': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'name': 'hosted_datacenter', 'version': '3', 'domains': '3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 1}, 'dominfo': {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'status': 'Active', 'diskfree': '75295293440', 'isoprefix': '', 'alerts': [], 'disktotal': '422943129600', 'version': 3}}} >Thread-27::DEBUG::2014-01-22 16:50:32,760::task::579::TaskManager.Task::(_updateState) Task=`6ef8fc04-c8dc-4963-99ad-e2f4cc98166f`::moving from state preparing -> state finished >Thread-27::DEBUG::2014-01-22 16:50:32,760::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-27::DEBUG::2014-01-22 16:50:32,760::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27::DEBUG::2014-01-22 16:50:32,760::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-27::DEBUG::2014-01-22 16:50:32,760::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-27::DEBUG::2014-01-22 16:50:32,760::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-27::DEBUG::2014-01-22 16:50:32,760::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-27::DEBUG::2014-01-22 16:50:32,760::task::974::TaskManager.Task::(_decref) Task=`6ef8fc04-c8dc-4963-99ad-e2f4cc98166f`::ref 0 aborting False >Thread-28::DEBUG::2014-01-22 16:50:32,812::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-28::DEBUG::2014-01-22 16:50:32,812::task::579::TaskManager.Task::(_updateState) Task=`3ec0d4c5-4be2-4a37-8803-7eed795c6178`::moving from state init -> state preparing >Thread-28::INFO::2014-01-22 16:50:32,813::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-01-22 16:50:32,813::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000345585', 'lastCheck': '0.3', 'valid': True}} >Thread-28::DEBUG::2014-01-22 16:50:32,813::task::1168::TaskManager.Task::(prepare) Task=`3ec0d4c5-4be2-4a37-8803-7eed795c6178`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000345585', 'lastCheck': '0.3', 'valid': True}} >Thread-28::DEBUG::2014-01-22 16:50:32,813::task::579::TaskManager.Task::(_updateState) Task=`3ec0d4c5-4be2-4a37-8803-7eed795c6178`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-01-22 16:50:32,814::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-01-22 16:50:32,814::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-01-22 16:50:32,814::task::974::TaskManager.Task::(_decref) Task=`3ec0d4c5-4be2-4a37-8803-7eed795c6178`::ref 0 aborting False >Thread-29::DEBUG::2014-01-22 16:50:33,033::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-29::DEBUG::2014-01-22 16:50:33,034::task::579::TaskManager.Task::(_updateState) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::moving from state init -> state preparing >Thread-29::INFO::2014-01-22 16:50:33,034::logUtils::44::dispatcher::(wrapper) Run and protect: createVolume(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', size=52428800, volFormat=5, preallocate=2, diskType=2, volUUID='fafd2cd2-6294-465b-b626-de8019782459', desc='Hosted Engine Image', srcImgUUID='00000000-0000-0000-0000-000000000000', srcVolUUID='00000000-0000-0000-0000-000000000000') >Thread-29::DEBUG::2014-01-22 16:50:33,035::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`221142e4-340e-481d-948b-0843cf215346`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1484' at 'createVolume' >Thread-29::DEBUG::2014-01-22 16:50:33,035::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'shared' >Thread-29::DEBUG::2014-01-22 16:50:33,036::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'shared' (1 active user) >Thread-29::DEBUG::2014-01-22 16:50:33,036::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`221142e4-340e-481d-948b-0843cf215346`::Granted request >Thread-29::DEBUG::2014-01-22 16:50:33,036::task::811::TaskManager.Task::(resourceAcquired) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (shared) >Thread-29::DEBUG::2014-01-22 16:50:33,036::task::974::TaskManager.Task::(_decref) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::ref 1 aborting False >Thread-29::DEBUG::2014-01-22 16:50:33,042::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >Thread-29::DEBUG::2014-01-22 16:50:33,056::taskManager::68::TaskManager::(scheduleJob) scheduled job createVolume for task b7b46b89-f92d-413e-afb2-88d354fe8c64 >Thread-29::INFO::2014-01-22 16:50:33,056::logUtils::47::dispatcher::(wrapper) Run and protect: createVolume, Return response: None >Thread-29::DEBUG::2014-01-22 16:50:33,056::task::1163::TaskManager.Task::(prepare) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::Prepare: 1 jobs exist, move to acquiring >Thread-29::DEBUG::2014-01-22 16:50:33,056::task::579::TaskManager.Task::(_updateState) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::moving from state preparing -> state acquiring >Thread-29::DEBUG::2014-01-22 16:50:33,057::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >Thread-29::DEBUG::2014-01-22 16:50:33,072::task::579::TaskManager.Task::(_updateState) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::moving from state acquiring -> state queued >Thread-29::DEBUG::2014-01-22 16:50:33,073::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >Thread-29::DEBUG::2014-01-22 16:50:33,089::taskManager::50::TaskManager::(_queueTask) queuing task: b7b46b89-f92d-413e-afb2-88d354fe8c64 >Thread-29::DEBUG::2014-01-22 16:50:33,090::taskManager::56::TaskManager::(_queueTask) task queued: b7b46b89-f92d-413e-afb2-88d354fe8c64 >Thread-2::DEBUG::2014-01-22 16:50:33,090::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-29::DEBUG::2014-01-22 16:50:33,091::task::1165::TaskManager.Task::(prepare) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::returning >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,092::threadPool::205::Misc.ThreadPool.WorkerThread::(run) Task: b7b46b89-f92d-413e-afb2-88d354fe8c64 running: <bound method Task.commit of <storage.task.Task instance at 0x7f7a500997e8>> with: None >Thread-29::DEBUG::2014-01-22 16:50:33,093::task::974::TaskManager.Task::(_decref) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::ref 0 aborting False >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,094::task::1176::TaskManager.Task::(commit) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::committing task: b7b46b89-f92d-413e-afb2-88d354fe8c64 >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,095::task::579::TaskManager.Task::(_updateState) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::moving from state queued -> state running >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,098::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,118::task::889::TaskManager.Task::(_runJobs) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::Task.run: running job 0: createVolume: <bound method StoragePool.createVolume of <storage.sp.StoragePool object at 0x7f7a500bbed0>> (args: ('3900e952-713e-4ea9-b32c-097cd4994bc5', 'e836fc1b-612b-447a-a52a-60db85dd9699', 52428800, 5, 2, 2, 'fafd2cd2-6294-465b-b626-de8019782459', 'Hosted Engine Image', '00000000-0000-0000-0000-000000000000', '00000000-0000-0000-0000-000000000000') kwargs: {}) >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,119::task::315::TaskManager.Task::(run) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::Job.run: running createVolume: <bound method StoragePool.createVolume of <storage.sp.StoragePool object at 0x7f7a500bbed0>> (args: ('3900e952-713e-4ea9-b32c-097cd4994bc5', 'e836fc1b-612b-447a-a52a-60db85dd9699', 52428800, 5, 2, 2, 'fafd2cd2-6294-465b-b626-de8019782459', 'Hosted Engine Image', '00000000-0000-0000-0000-000000000000', '00000000-0000-0000-0000-000000000000') kwargs: {}) callback None >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,119::resourceManager::197::ResourceManager.Request::(__init__) ResName=`3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.e836fc1b-612b-447a-a52a-60db85dd9699`ReqID=`b5218e0a-283e-497e-9023-555a19e49294`::Request was made in '/usr/share/vdsm/storage/sp.py' line '1895' at 'createVolume' >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,120::resourceManager::541::ResourceManager::(registerResource) Trying to register resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.e836fc1b-612b-447a-a52a-60db85dd9699' for lock type 'exclusive' >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,121::resourceFactories::125::Storage.ResourcesFactories::(__getResourceCandidatesList) Image e836fc1b-612b-447a-a52a-60db85dd9699 does not exist in domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,121::resourceManager::600::ResourceManager::(registerResource) Resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.e836fc1b-612b-447a-a52a-60db85dd9699' is free. Now locking as 'exclusive' (1 active user) >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,121::resourceManager::237::ResourceManager.Request::(grant) ResName=`3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.e836fc1b-612b-447a-a52a-60db85dd9699`ReqID=`b5218e0a-283e-497e-9023-555a19e49294`::Granted request >b7b46b89-f92d-413e-afb2-88d354fe8c64::INFO::2014-01-22 16:50:33,122::image::123::Storage.Image::(create) Create placeholder /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699 for image's volumes >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,123::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >b7b46b89-f92d-413e-afb2-88d354fe8c64::INFO::2014-01-22 16:50:33,143::volume::446::Storage.Volume::(create) Creating volume fafd2cd2-6294-465b-b626-de8019782459 >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,143::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >Thread-30::DEBUG::2014-01-22 16:50:33,151::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-30::DEBUG::2014-01-22 16:50:33,153::task::579::TaskManager.Task::(_updateState) Task=`143af865-7ad6-48e9-847e-843d34ba7411`::moving from state init -> state preparing >Thread-30::INFO::2014-01-22 16:50:33,153::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-30::DEBUG::2014-01-22 16:50:33,154::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-30::DEBUG::2014-01-22 16:50:33,154::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b7b46b89-f92d-413e-afb2-88d354fe8c64 >Thread-30::DEBUG::2014-01-22 16:50:33,154::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b7b46b89-f92d-413e-afb2-88d354fe8c64'} >Thread-30::DEBUG::2014-01-22 16:50:33,154::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {'b7b46b89-f92d-413e-afb2-88d354fe8c64': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b7b46b89-f92d-413e-afb2-88d354fe8c64'}} >Thread-30::INFO::2014-01-22 16:50:33,155::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {'b7b46b89-f92d-413e-afb2-88d354fe8c64': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b7b46b89-f92d-413e-afb2-88d354fe8c64'}}} >Thread-30::DEBUG::2014-01-22 16:50:33,155::task::1168::TaskManager.Task::(prepare) Task=`143af865-7ad6-48e9-847e-843d34ba7411`::finished: {'allTasksStatus': {'b7b46b89-f92d-413e-afb2-88d354fe8c64': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b7b46b89-f92d-413e-afb2-88d354fe8c64'}}} >Thread-30::DEBUG::2014-01-22 16:50:33,155::task::579::TaskManager.Task::(_updateState) Task=`143af865-7ad6-48e9-847e-843d34ba7411`::moving from state preparing -> state finished >Thread-30::DEBUG::2014-01-22 16:50:33,157::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30::DEBUG::2014-01-22 16:50:33,158::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30::DEBUG::2014-01-22 16:50:33,158::task::974::TaskManager.Task::(_decref) Task=`143af865-7ad6-48e9-847e-843d34ba7411`::ref 0 aborting False >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,171::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >b7b46b89-f92d-413e-afb2-88d354fe8c64::INFO::2014-01-22 16:50:33,213::fileVolume::151::Storage.Volume::(_create) Request to create RAW volume /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459 with size = 52428800 sectors >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,216::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,250::fileVolume::382::Storage.Volume::(newVolumeLease) Initializing volume lease volUUID=fafd2cd2-6294-465b-b626-de8019782459 sdUUID=3900e952-713e-4ea9-b32c-097cd4994bc5, metaId=('/rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459',) >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,274::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,300::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.e836fc1b-612b-447a-a52a-60db85dd9699' >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,301::resourceManager::634::ResourceManager::(releaseResource) Released resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.e836fc1b-612b-447a-a52a-60db85dd9699' (0 active users) >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,301::resourceManager::640::ResourceManager::(releaseResource) Resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.e836fc1b-612b-447a-a52a-60db85dd9699' is free, finding out if anyone is waiting for it. >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,301::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource '3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS.e836fc1b-612b-447a-a52a-60db85dd9699', Clearing records. >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,301::task::579::TaskManager.Task::(_updateState) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::moving from state running -> state finished >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,302::task::736::TaskManager.Task::(_save) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::_save: orig /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64 temp /rhev/data-center/e1263145-a6a9-4993-9d8a-cd92a3e5c1fb/mastersd/master/tasks/b7b46b89-f92d-413e-afb2-88d354fe8c64.temp >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,328::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>} >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,329::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,329::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,329::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,330::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,330::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,330::task::897::TaskManager.Task::(_runJobs) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::Task.run: exit - success: result {'uuid': 'fafd2cd2-6294-465b-b626-de8019782459'} >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,330::task::974::TaskManager.Task::(_decref) Task=`b7b46b89-f92d-413e-afb2-88d354fe8c64`::ref 0 aborting False >b7b46b89-f92d-413e-afb2-88d354fe8c64::DEBUG::2014-01-22 16:50:33,331::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-31::DEBUG::2014-01-22 16:50:34,211::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-31::DEBUG::2014-01-22 16:50:34,211::task::579::TaskManager.Task::(_updateState) Task=`98724245-d20e-43f5-a916-4ae08bf858aa`::moving from state init -> state preparing >Thread-31::INFO::2014-01-22 16:50:34,212::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-31::DEBUG::2014-01-22 16:50:34,212::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-31::DEBUG::2014-01-22 16:50:34,212::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b7b46b89-f92d-413e-afb2-88d354fe8c64 >Thread-31::DEBUG::2014-01-22 16:50:34,212::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b7b46b89-f92d-413e-afb2-88d354fe8c64'} >Thread-31::DEBUG::2014-01-22 16:50:34,213::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {'b7b46b89-f92d-413e-afb2-88d354fe8c64': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b7b46b89-f92d-413e-afb2-88d354fe8c64'}} >Thread-31::INFO::2014-01-22 16:50:34,213::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {'b7b46b89-f92d-413e-afb2-88d354fe8c64': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b7b46b89-f92d-413e-afb2-88d354fe8c64'}}} >Thread-31::DEBUG::2014-01-22 16:50:34,213::task::1168::TaskManager.Task::(prepare) Task=`98724245-d20e-43f5-a916-4ae08bf858aa`::finished: {'allTasksStatus': {'b7b46b89-f92d-413e-afb2-88d354fe8c64': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b7b46b89-f92d-413e-afb2-88d354fe8c64'}}} >Thread-31::DEBUG::2014-01-22 16:50:34,213::task::579::TaskManager.Task::(_updateState) Task=`98724245-d20e-43f5-a916-4ae08bf858aa`::moving from state preparing -> state finished >Thread-31::DEBUG::2014-01-22 16:50:34,214::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31::DEBUG::2014-01-22 16:50:34,214::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31::DEBUG::2014-01-22 16:50:34,214::task::974::TaskManager.Task::(_decref) Task=`98724245-d20e-43f5-a916-4ae08bf858aa`::ref 0 aborting False >Thread-32::DEBUG::2014-01-22 16:50:34,265::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-32::DEBUG::2014-01-22 16:50:34,265::task::579::TaskManager.Task::(_updateState) Task=`715885de-aba5-4e80-9098-5cfc5c0167c9`::moving from state init -> state preparing >Thread-32::INFO::2014-01-22 16:50:34,266::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='b7b46b89-f92d-413e-afb2-88d354fe8c64', spUUID=None, options=None) >Thread-32::DEBUG::2014-01-22 16:50:34,266::taskManager::161::TaskManager::(clearTask) Entry. taskID: b7b46b89-f92d-413e-afb2-88d354fe8c64 >Thread-32::DEBUG::2014-01-22 16:50:34,274::taskManager::166::TaskManager::(clearTask) Return. >Thread-32::INFO::2014-01-22 16:50:34,274::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-32::DEBUG::2014-01-22 16:50:34,275::task::1168::TaskManager.Task::(prepare) Task=`715885de-aba5-4e80-9098-5cfc5c0167c9`::finished: None >Thread-32::DEBUG::2014-01-22 16:50:34,275::task::579::TaskManager.Task::(_updateState) Task=`715885de-aba5-4e80-9098-5cfc5c0167c9`::moving from state preparing -> state finished >Thread-32::DEBUG::2014-01-22 16:50:34,275::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32::DEBUG::2014-01-22 16:50:34,275::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32::DEBUG::2014-01-22 16:50:34,275::task::974::TaskManager.Task::(_decref) Task=`715885de-aba5-4e80-9098-5cfc5c0167c9`::ref 0 aborting False >Thread-33::DEBUG::2014-01-22 16:50:34,325::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-33::DEBUG::2014-01-22 16:50:34,325::task::579::TaskManager.Task::(_updateState) Task=`d825cfc7-ef87-4748-a0ae-420e62bcfabb`::moving from state init -> state preparing >Thread-33::INFO::2014-01-22 16:50:34,326::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-33::DEBUG::2014-01-22 16:50:34,326::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-33::DEBUG::2014-01-22 16:50:34,326::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-33::INFO::2014-01-22 16:50:34,326::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-33::DEBUG::2014-01-22 16:50:34,326::task::1168::TaskManager.Task::(prepare) Task=`d825cfc7-ef87-4748-a0ae-420e62bcfabb`::finished: {'allTasksStatus': {}} >Thread-33::DEBUG::2014-01-22 16:50:34,327::task::579::TaskManager.Task::(_updateState) Task=`d825cfc7-ef87-4748-a0ae-420e62bcfabb`::moving from state preparing -> state finished >Thread-33::DEBUG::2014-01-22 16:50:34,327::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33::DEBUG::2014-01-22 16:50:34,327::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33::DEBUG::2014-01-22 16:50:34,327::task::974::TaskManager.Task::(_decref) Task=`d825cfc7-ef87-4748-a0ae-420e62bcfabb`::ref 0 aborting False >Thread-34::DEBUG::2014-01-22 16:50:34,378::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-34::DEBUG::2014-01-22 16:50:34,378::task::579::TaskManager.Task::(_updateState) Task=`c53df5e2-028a-4396-adad-54fd6a7367c9`::moving from state init -> state preparing >Thread-34::INFO::2014-01-22 16:50:34,379::logUtils::44::dispatcher::(wrapper) Run and protect: spmStop(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', options=None) >Thread-34::DEBUG::2014-01-22 16:50:34,379::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`3d9138b7-893c-4fee-9a2e-eb168b4491b0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '615' at 'spmStop' >Thread-34::DEBUG::2014-01-22 16:50:34,379::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-34::DEBUG::2014-01-22 16:50:34,380::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-34::DEBUG::2014-01-22 16:50:34,380::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`3d9138b7-893c-4fee-9a2e-eb168b4491b0`::Granted request >Thread-34::DEBUG::2014-01-22 16:50:34,380::task::811::TaskManager.Task::(resourceAcquired) Task=`c53df5e2-028a-4396-adad-54fd6a7367c9`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-34::DEBUG::2014-01-22 16:50:34,381::task::974::TaskManager.Task::(_decref) Task=`c53df5e2-028a-4396-adad-54fd6a7367c9`::ref 1 aborting False >Thread-34::DEBUG::2014-01-22 16:50:34,381::sp::343::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process >Thread-34::DEBUG::2014-01-22 16:50:34,381::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`cab2879b-7b3f-4f75-bfd5-5405c24578f7`::Request was made in '/usr/share/vdsm/storage/sp.py' line '345' at '_shutDownUpgrade' >Thread-34::DEBUG::2014-01-22 16:50:34,382::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-34::DEBUG::2014-01-22 16:50:34,382::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-34::DEBUG::2014-01-22 16:50:34,382::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`cab2879b-7b3f-4f75-bfd5-5405c24578f7`::Granted request >Thread-34::DEBUG::2014-01-22 16:50:34,383::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-34::DEBUG::2014-01-22 16:50:34,383::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-34::DEBUG::2014-01-22 16:50:34,383::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-34::DEBUG::2014-01-22 16:50:34,383::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-34::DEBUG::2014-01-22 16:50:34,384::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-34::DEBUG::2014-01-22 16:50:34,384::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-34::DEBUG::2014-01-22 16:50:34,385::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=e8504e783bbf704c2dbec50f6a1b369b8550bd07'] >Thread-34::DEBUG::2014-01-22 16:50:34,390::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-34::INFO::2014-01-22 16:50:34,390::clusterlock::255::SANLock::(release) Releasing cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-34::DEBUG::2014-01-22 16:50:34,393::clusterlock::266::SANLock::(release) Cluster lock for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully released >Thread-34::INFO::2014-01-22 16:50:34,393::logUtils::47::dispatcher::(wrapper) Run and protect: spmStop, Return response: None >Thread-34::DEBUG::2014-01-22 16:50:34,393::task::1168::TaskManager.Task::(prepare) Task=`c53df5e2-028a-4396-adad-54fd6a7367c9`::finished: None >Thread-34::DEBUG::2014-01-22 16:50:34,393::task::579::TaskManager.Task::(_updateState) Task=`c53df5e2-028a-4396-adad-54fd6a7367c9`::moving from state preparing -> state finished >Thread-34::DEBUG::2014-01-22 16:50:34,394::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-34::DEBUG::2014-01-22 16:50:34,394::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34::DEBUG::2014-01-22 16:50:34,394::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-34::DEBUG::2014-01-22 16:50:34,394::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-34::DEBUG::2014-01-22 16:50:34,395::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-34::DEBUG::2014-01-22 16:50:34,395::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-34::DEBUG::2014-01-22 16:50:34,395::task::974::TaskManager.Task::(_decref) Task=`c53df5e2-028a-4396-adad-54fd6a7367c9`::ref 0 aborting False >Thread-35::DEBUG::2014-01-22 16:50:34,446::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-35::DEBUG::2014-01-22 16:50:34,446::task::579::TaskManager.Task::(_updateState) Task=`8fc45ff0-c366-47dd-8bd0-ae537d0b3ce8`::moving from state init -> state preparing >Thread-35::INFO::2014-01-22 16:50:34,447::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', hostID=1, scsiKey='e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', remove=False, options=None) >Thread-35::DEBUG::2014-01-22 16:50:34,447::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`86f73e1d-06d5-45d1-b110-b4c99f8a5406`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1111' at 'disconnectStoragePool' >Thread-35::DEBUG::2014-01-22 16:50:34,448::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' for lock type 'exclusive' >Thread-35::DEBUG::2014-01-22 16:50:34,448::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free. Now locking as 'exclusive' (1 active user) >Thread-35::DEBUG::2014-01-22 16:50:34,448::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb`ReqID=`86f73e1d-06d5-45d1-b110-b4c99f8a5406`::Granted request >Thread-35::DEBUG::2014-01-22 16:50:34,448::task::811::TaskManager.Task::(resourceAcquired) Task=`8fc45ff0-c366-47dd-8bd0-ae537d0b3ce8`::_resourcesAcquired: Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb (exclusive) >Thread-35::DEBUG::2014-01-22 16:50:34,448::task::974::TaskManager.Task::(_decref) Task=`8fc45ff0-c366-47dd-8bd0-ae537d0b3ce8`::ref 1 aborting False >Thread-35::DEBUG::2014-01-22 16:50:34,448::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`dfd1935b-8613-4635-bc2d-e79452c1eddb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1119' at '_disconnectPool' >Thread-35::DEBUG::2014-01-22 16:50:34,449::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-35::DEBUG::2014-01-22 16:50:34,449::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-35::DEBUG::2014-01-22 16:50:34,449::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`dfd1935b-8613-4635-bc2d-e79452c1eddb`::Granted request >Thread-35::INFO::2014-01-22 16:50:34,449::sp::711::Storage.StoragePool::(disconnect) Disconnect from the storage pool e1263145-a6a9-4993-9d8a-cd92a3e5c1fb >Thread-35::INFO::2014-01-22 16:50:34,449::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-18::DEBUG::2014-01-22 16:50:34,449::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-18::INFO::2014-01-22 16:50:34,450::clusterlock::197::SANLock::(releaseHostId) Releasing host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-18::DEBUG::2014-01-22 16:50:37,451::clusterlock::207::SANLock::(releaseHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 released successfully (id: 1) >Thread-35::DEBUG::2014-01-22 16:50:37,452::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-35::DEBUG::2014-01-22 16:50:37,453::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-35::DEBUG::2014-01-22 16:50:37,453::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-35::DEBUG::2014-01-22 16:50:37,453::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-35::INFO::2014-01-22 16:50:37,453::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: True >Thread-35::DEBUG::2014-01-22 16:50:37,454::task::1168::TaskManager.Task::(prepare) Task=`8fc45ff0-c366-47dd-8bd0-ae537d0b3ce8`::finished: True >Thread-35::DEBUG::2014-01-22 16:50:37,454::task::579::TaskManager.Task::(_updateState) Task=`8fc45ff0-c366-47dd-8bd0-ae537d0b3ce8`::moving from state preparing -> state finished >Thread-35::DEBUG::2014-01-22 16:50:37,454::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb': < ResourceRef 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', isValid: 'True' obj: 'None'>} >Thread-35::DEBUG::2014-01-22 16:50:37,454::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35::DEBUG::2014-01-22 16:50:37,455::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' >Thread-35::DEBUG::2014-01-22 16:50:37,455::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' (0 active users) >Thread-35::DEBUG::2014-01-22 16:50:37,455::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb' is free, finding out if anyone is waiting for it. >Thread-35::DEBUG::2014-01-22 16:50:37,455::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', Clearing records. >Thread-35::DEBUG::2014-01-22 16:50:37,456::task::974::TaskManager.Task::(_decref) Task=`8fc45ff0-c366-47dd-8bd0-ae537d0b3ce8`::ref 0 aborting False >Thread-36::DEBUG::2014-01-22 16:50:37,508::BindingXMLRPC::177::vds::(wrapper) client [127.0.0.1] >Thread-36::DEBUG::2014-01-22 16:50:37,508::task::579::TaskManager.Task::(_updateState) Task=`9df20dd2-ec3a-430d-abe3-b341942f6c5b`::moving from state init -> state preparing >Thread-36::INFO::2014-01-22 16:50:37,509::logUtils::44::dispatcher::(wrapper) Run and protect: startMonitoringDomain(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', hostID=1, options=None) >Thread-36::DEBUG::2014-01-22 16:50:37,509::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`2a3e31ba-656c-4cb9-8800-3d583edf3e96`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3694' at 'startMonitoringDomain' >Thread-36::DEBUG::2014-01-22 16:50:37,509::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-36::DEBUG::2014-01-22 16:50:37,510::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-36::DEBUG::2014-01-22 16:50:37,510::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`2a3e31ba-656c-4cb9-8800-3d583edf3e96`::Granted request >Thread-36::INFO::2014-01-22 16:50:37,510::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-37::DEBUG::2014-01-22 16:50:37,511::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-36::DEBUG::2014-01-22 16:50:37,511::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-36::DEBUG::2014-01-22 16:50:37,512::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-36::DEBUG::2014-01-22 16:50:37,512::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-36::DEBUG::2014-01-22 16:50:37,512::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-36::INFO::2014-01-22 16:50:37,513::logUtils::47::dispatcher::(wrapper) Run and protect: startMonitoringDomain, Return response: None >Thread-36::DEBUG::2014-01-22 16:50:37,514::task::1168::TaskManager.Task::(prepare) Task=`9df20dd2-ec3a-430d-abe3-b341942f6c5b`::finished: None >Thread-36::DEBUG::2014-01-22 16:50:37,514::task::579::TaskManager.Task::(_updateState) Task=`9df20dd2-ec3a-430d-abe3-b341942f6c5b`::moving from state preparing -> state finished >Thread-36::DEBUG::2014-01-22 16:50:37,516::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-36::DEBUG::2014-01-22 16:50:37,516::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-36::DEBUG::2014-01-22 16:50:37,516::task::974::TaskManager.Task::(_decref) Task=`9df20dd2-ec3a-430d-abe3-b341942f6c5b`::ref 0 aborting False >Thread-37::DEBUG::2014-01-22 16:50:37,523::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:50:37,530::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000394348 s, 1.2 MB/s\n'; <rc> = 0 >Thread-37::DEBUG::2014-01-22 16:50:37,535::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3900e952-713e-4ea9-b32c-097cd4994bc5 changed its status to Valid >Thread-38::DEBUG::2014-01-22 16:50:37,535::misc::931::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-37::INFO::2014-01-22 16:50:37,536::clusterlock::174::SANLock::(acquireHostId) Acquiring host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 (id: 1) >Thread-38::DEBUG::2014-01-22 16:50:37,537::misc::941::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-37::DEBUG::2014-01-22 16:50:37,537::clusterlock::192::SANLock::(acquireHostId) Host id for domain 3900e952-713e-4ea9-b32c-097cd4994bc5 successfully acquired (id: 1) >Thread-38::DEBUG::2014-01-22 16:50:37,539::misc::951::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-39::INFO::2014-01-22 16:50:37,540::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-40::DEBUG::2014-01-22 16:50:42,575::task::579::TaskManager.Task::(_updateState) Task=`69806570-4f70-4932-9e2b-f0be4f1d8222`::moving from state init -> state preparing >Thread-40::INFO::2014-01-22 16:50:42,575::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-40::INFO::2014-01-22 16:50:42,575::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000394348', 'lastCheck': '5.0', 'valid': True}} >Thread-40::DEBUG::2014-01-22 16:50:42,576::task::1168::TaskManager.Task::(prepare) Task=`69806570-4f70-4932-9e2b-f0be4f1d8222`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000394348', 'lastCheck': '5.0', 'valid': True}} >Thread-40::DEBUG::2014-01-22 16:50:42,576::task::579::TaskManager.Task::(_updateState) Task=`69806570-4f70-4932-9e2b-f0be4f1d8222`::moving from state preparing -> state finished >Thread-40::DEBUG::2014-01-22 16:50:42,576::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-40::DEBUG::2014-01-22 16:50:42,576::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-40::DEBUG::2014-01-22 16:50:42,577::task::974::TaskManager.Task::(_decref) Task=`69806570-4f70-4932-9e2b-f0be4f1d8222`::ref 0 aborting False >Thread-40::ERROR::2014-01-22 16:50:42,578::brokerlink::72::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(connect) Failed to connect to broker: [Errno 2] No such file or directory >Thread-40::ERROR::2014-01-22 16:50:42,579::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 197, in get_local_host_score > with broker.connection(): > File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__ > return self.gen.next() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 96, in connection > self.connect() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 64, in connect > self._socket.connect(constants.BROKER_SOCKET_FILE) > File "<string>", line 1, in connect >error: [Errno 2] No such file or directory >Thread-37::DEBUG::2014-01-22 16:50:47,551::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:50:47,558::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000384901 s, 1.2 MB/s\n'; <rc> = 0 >Thread-41::DEBUG::2014-01-22 16:50:47,652::task::579::TaskManager.Task::(_updateState) Task=`0514ffcc-8095-45a2-a6de-920eec714bb3`::moving from state init -> state preparing >Thread-41::INFO::2014-01-22 16:50:47,652::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-41::INFO::2014-01-22 16:50:47,652::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000384901', 'lastCheck': '0.1', 'valid': True}} >Thread-41::DEBUG::2014-01-22 16:50:47,652::task::1168::TaskManager.Task::(prepare) Task=`0514ffcc-8095-45a2-a6de-920eec714bb3`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000384901', 'lastCheck': '0.1', 'valid': True}} >Thread-41::DEBUG::2014-01-22 16:50:47,653::task::579::TaskManager.Task::(_updateState) Task=`0514ffcc-8095-45a2-a6de-920eec714bb3`::moving from state preparing -> state finished >Thread-41::DEBUG::2014-01-22 16:50:47,653::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-41::DEBUG::2014-01-22 16:50:47,653::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-41::DEBUG::2014-01-22 16:50:47,653::task::974::TaskManager.Task::(_decref) Task=`0514ffcc-8095-45a2-a6de-920eec714bb3`::ref 0 aborting False >Thread-41::ERROR::2014-01-22 16:50:47,655::brokerlink::72::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(connect) Failed to connect to broker: [Errno 2] No such file or directory >Thread-41::ERROR::2014-01-22 16:50:47,655::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 197, in get_local_host_score > with broker.connection(): > File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__ > return self.gen.next() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 96, in connection > self.connect() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 64, in connect > self._socket.connect(constants.BROKER_SOCKET_FILE) > File "<string>", line 1, in connect >error: [Errno 2] No such file or directory >Thread-42::DEBUG::2014-01-22 16:50:52,721::task::579::TaskManager.Task::(_updateState) Task=`7d25bc25-6c36-4135-ae28-e14be26f7c33`::moving from state init -> state preparing >Thread-42::INFO::2014-01-22 16:50:52,721::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-42::INFO::2014-01-22 16:50:52,721::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000384901', 'lastCheck': '5.2', 'valid': True}} >Thread-42::DEBUG::2014-01-22 16:50:52,722::task::1168::TaskManager.Task::(prepare) Task=`7d25bc25-6c36-4135-ae28-e14be26f7c33`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000384901', 'lastCheck': '5.2', 'valid': True}} >Thread-42::DEBUG::2014-01-22 16:50:52,722::task::579::TaskManager.Task::(_updateState) Task=`7d25bc25-6c36-4135-ae28-e14be26f7c33`::moving from state preparing -> state finished >Thread-42::DEBUG::2014-01-22 16:50:52,722::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-42::DEBUG::2014-01-22 16:50:52,722::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-42::DEBUG::2014-01-22 16:50:52,722::task::974::TaskManager.Task::(_decref) Task=`7d25bc25-6c36-4135-ae28-e14be26f7c33`::ref 0 aborting False >Thread-42::ERROR::2014-01-22 16:50:52,724::brokerlink::72::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(connect) Failed to connect to broker: [Errno 2] No such file or directory >Thread-42::ERROR::2014-01-22 16:50:52,724::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 197, in get_local_host_score > with broker.connection(): > File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__ > return self.gen.next() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 96, in connection > self.connect() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 64, in connect > self._socket.connect(constants.BROKER_SOCKET_FILE) > File "<string>", line 1, in connect >error: [Errno 2] No such file or directory >Thread-37::DEBUG::2014-01-22 16:50:57,573::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:50:57,580::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000403537 s, 1.2 MB/s\n'; <rc> = 0 >Thread-43::DEBUG::2014-01-22 16:50:57,789::task::579::TaskManager.Task::(_updateState) Task=`079162c6-2769-4a0e-b4e2-0255cd26ce79`::moving from state init -> state preparing >Thread-43::INFO::2014-01-22 16:50:57,789::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-43::INFO::2014-01-22 16:50:57,789::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000403537', 'lastCheck': '0.2', 'valid': True}} >Thread-43::DEBUG::2014-01-22 16:50:57,790::task::1168::TaskManager.Task::(prepare) Task=`079162c6-2769-4a0e-b4e2-0255cd26ce79`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000403537', 'lastCheck': '0.2', 'valid': True}} >Thread-43::DEBUG::2014-01-22 16:50:57,790::task::579::TaskManager.Task::(_updateState) Task=`079162c6-2769-4a0e-b4e2-0255cd26ce79`::moving from state preparing -> state finished >Thread-43::DEBUG::2014-01-22 16:50:57,790::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-43::DEBUG::2014-01-22 16:50:57,790::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-43::DEBUG::2014-01-22 16:50:57,790::task::974::TaskManager.Task::(_decref) Task=`079162c6-2769-4a0e-b4e2-0255cd26ce79`::ref 0 aborting False >Thread-43::ERROR::2014-01-22 16:50:57,792::brokerlink::72::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(connect) Failed to connect to broker: [Errno 2] No such file or directory >Thread-43::ERROR::2014-01-22 16:50:57,792::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 197, in get_local_host_score > with broker.connection(): > File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__ > return self.gen.next() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 96, in connection > self.connect() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 64, in connect > self._socket.connect(constants.BROKER_SOCKET_FILE) > File "<string>", line 1, in connect >error: [Errno 2] No such file or directory >Thread-44::DEBUG::2014-01-22 16:51:02,858::task::579::TaskManager.Task::(_updateState) Task=`a9623e77-1253-41e9-b59d-764515804f58`::moving from state init -> state preparing >Thread-44::INFO::2014-01-22 16:51:02,858::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-44::INFO::2014-01-22 16:51:02,858::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000403537', 'lastCheck': '5.3', 'valid': True}} >Thread-44::DEBUG::2014-01-22 16:51:02,858::task::1168::TaskManager.Task::(prepare) Task=`a9623e77-1253-41e9-b59d-764515804f58`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000403537', 'lastCheck': '5.3', 'valid': True}} >Thread-44::DEBUG::2014-01-22 16:51:02,858::task::579::TaskManager.Task::(_updateState) Task=`a9623e77-1253-41e9-b59d-764515804f58`::moving from state preparing -> state finished >Thread-44::DEBUG::2014-01-22 16:51:02,858::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-44::DEBUG::2014-01-22 16:51:02,859::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-44::DEBUG::2014-01-22 16:51:02,859::task::974::TaskManager.Task::(_decref) Task=`a9623e77-1253-41e9-b59d-764515804f58`::ref 0 aborting False >Thread-44::ERROR::2014-01-22 16:51:02,859::brokerlink::72::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(connect) Failed to connect to broker: [Errno 2] No such file or directory >Thread-44::ERROR::2014-01-22 16:51:02,859::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 197, in get_local_host_score > with broker.connection(): > File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__ > return self.gen.next() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 96, in connection > self.connect() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 64, in connect > self._socket.connect(constants.BROKER_SOCKET_FILE) > File "<string>", line 1, in connect >error: [Errno 2] No such file or directory >Thread-37::DEBUG::2014-01-22 16:51:07,593::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:51:07,600::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.0003701 s, 1.3 MB/s\n'; <rc> = 0 >Thread-45::DEBUG::2014-01-22 16:51:07,923::task::579::TaskManager.Task::(_updateState) Task=`c27f346f-c27b-4880-b75a-2e59e6efcb32`::moving from state init -> state preparing >Thread-45::INFO::2014-01-22 16:51:07,923::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-45::INFO::2014-01-22 16:51:07,923::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003701', 'lastCheck': '0.3', 'valid': True}} >Thread-45::DEBUG::2014-01-22 16:51:07,924::task::1168::TaskManager.Task::(prepare) Task=`c27f346f-c27b-4880-b75a-2e59e6efcb32`::finished: {'3900e952-713e-4ea9-b32c-097cd4994bc5': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003701', 'lastCheck': '0.3', 'valid': True}} >Thread-45::DEBUG::2014-01-22 16:51:07,924::task::579::TaskManager.Task::(_updateState) Task=`c27f346f-c27b-4880-b75a-2e59e6efcb32`::moving from state preparing -> state finished >Thread-45::DEBUG::2014-01-22 16:51:07,924::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-45::DEBUG::2014-01-22 16:51:07,924::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-45::DEBUG::2014-01-22 16:51:07,924::task::974::TaskManager.Task::(_decref) Task=`c27f346f-c27b-4880-b75a-2e59e6efcb32`::ref 0 aborting False >Thread-45::ERROR::2014-01-22 16:51:07,926::brokerlink::72::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(connect) Failed to connect to broker: [Errno 2] No such file or directory >Thread-45::ERROR::2014-01-22 16:51:07,926::API::1223::vds::(getStats) failed to retrieve Hosted Engine HA score >Traceback (most recent call last): > File "/usr/share/vdsm/API.py", line 1221, in getStats > stats['haScore'] = haClient.HAClient().get_local_host_score() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 197, in get_local_host_score > with broker.connection(): > File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__ > return self.gen.next() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 96, in connection > self.connect() > File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 64, in connect > self._socket.connect(constants.BROKER_SOCKET_FILE) > File "<string>", line 1, in connect >error: [Errno 2] No such file or directory >Thread-46::DEBUG::2014-01-22 16:51:13,573::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-46::DEBUG::2014-01-22 16:51:13,580::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} >Thread-47::DEBUG::2014-01-22 16:51:13,719::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmCreate with ({'emulatedMachine': 'rhel6.5.0', 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'devices': [{'index': '2', 'iface': 'ide', 'specParams': {}, 'readonly': 'true', 'deviceId': 'ffb0a6bb-a46c-4f3b-a892-c3a05741aca3', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'index': '0', 'iface': 'virtio', 'format': 'raw', 'optional': 'false', 'poolID': '00000000-0000-0000-0000-000000000000', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'specParams': {}, 'readonly': 'false', 'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'deviceId': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'shared': 'exclusive', 'propagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:16:3e:4e:70:25', 'linkActive': 'true', 'network': 'rhevm', 'bootOrder': '1', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': 'c42e7bca-ecce-4890-8ad9-cf42acd2c9eb', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface'}, {'device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': '24b23aec-7638-4133-a866-da3935c7c5dc', 'alias': 'console0'}], 'smp': '2', 'memSize': '4096', 'cpuType': 'Westmere', 'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'vmName': 'HostedEngine', 'display': 'vnc'},) {} >Thread-47::INFO::2014-01-22 16:51:13,729::API::648::vds::(_getNetworkIp) network None: using 0 >Thread-47::INFO::2014-01-22 16:51:13,729::clientIF::394::vds::(createVm) vmContainerLock acquired by vm bfc3da60-a697-4890-aa90-6d293b6237dc >Thread-47::DEBUG::2014-01-22 16:51:13,730::libvirtconnection::131::libvirtconnection::(get) trying to connect libvirt >Thread-48::DEBUG::2014-01-22 16:51:13,777::vm::2115::vm.Vm::(_startUnderlyingVm) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Start >Thread-47::DEBUG::2014-01-22 16:51:13,777::clientIF::407::vds::(createVm) Total desktops after creation of bfc3da60-a697-4890-aa90-6d293b6237dc is 1 >Thread-47::DEBUG::2014-01-22 16:51:13,778::BindingXMLRPC::991::vds::(wrapper) return vmCreate with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'nicModel': 'rtl8139,pv', 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'emulatedMachine': 'rhel6.5.0', 'displayIp': '0', 'pid': '0', 'devices': [{'index': '2', 'iface': 'ide', 'specParams': {}, 'readonly': 'true', 'deviceId': 'ffb0a6bb-a46c-4f3b-a892-c3a05741aca3', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'index': '0', 'iface': 'virtio', 'format': 'raw', 'optional': 'false', 'poolID': '00000000-0000-0000-0000-000000000000', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'specParams': {}, 'readonly': 'false', 'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'deviceId': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'shared': 'exclusive', 'propagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:16:3e:4e:70:25', 'linkActive': 'true', 'network': 'rhevm', 'bootOrder': '1', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': 'c42e7bca-ecce-4890-8ad9-cf42acd2c9eb', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface'}, {'device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': '24b23aec-7638-4133-a866-da3935c7c5dc', 'alias': 'console0'}], 'smp': '2', 'vmType': 'kvm', 'displaySecurePort': '-1', 'memSize': '4096', 'displayPort': '-1', 'cpuType': 'Westmere', 'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'clientIp': '', 'vmName': 'HostedEngine', 'display': 'vnc'}} >Thread-48::DEBUG::2014-01-22 16:51:13,778::vm::2119::vm.Vm::(_startUnderlyingVm) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::_ongoingCreations acquired >Thread-48::INFO::2014-01-22 16:51:13,779::vm::2969::vm.Vm::(_run) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::VM wrapper has started >Thread-48::DEBUG::2014-01-22 16:51:13,780::task::579::TaskManager.Task::(_updateState) Task=`f0afe50f-d3f8-4ff2-b06c-2b506b90a694`::moving from state init -> state preparing >Thread-48::INFO::2014-01-22 16:51:13,780::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', volUUID='fafd2cd2-6294-465b-b626-de8019782459', options=None) >Thread-48::INFO::2014-01-22 16:51:13,784::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-48::DEBUG::2014-01-22 16:51:13,785::task::1168::TaskManager.Task::(prepare) Task=`f0afe50f-d3f8-4ff2-b06c-2b506b90a694`::finished: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-48::DEBUG::2014-01-22 16:51:13,785::task::579::TaskManager.Task::(_updateState) Task=`f0afe50f-d3f8-4ff2-b06c-2b506b90a694`::moving from state preparing -> state finished >Thread-48::DEBUG::2014-01-22 16:51:13,785::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-48::DEBUG::2014-01-22 16:51:13,785::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-48::DEBUG::2014-01-22 16:51:13,785::task::974::TaskManager.Task::(_decref) Task=`f0afe50f-d3f8-4ff2-b06c-2b506b90a694`::ref 0 aborting False >Thread-48::INFO::2014-01-22 16:51:13,786::clientIF::353::vds::(prepareVolumePath) prepared volume path: >Thread-48::DEBUG::2014-01-22 16:51:13,786::task::579::TaskManager.Task::(_updateState) Task=`52dbcf71-0da9-4d9e-8c3b-ec45d12f2b4b`::moving from state init -> state preparing >Thread-48::INFO::2014-01-22 16:51:13,786::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', leafUUID='fafd2cd2-6294-465b-b626-de8019782459') >Thread-48::DEBUG::2014-01-22 16:51:13,787::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`db40cf97-d0d8-4ab9-9a1f-85d2ff435f21`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3256' at 'prepareImage' >Thread-48::DEBUG::2014-01-22 16:51:13,787::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'shared' >Thread-48::DEBUG::2014-01-22 16:51:13,787::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'shared' (1 active user) >Thread-48::DEBUG::2014-01-22 16:51:13,787::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`db40cf97-d0d8-4ab9-9a1f-85d2ff435f21`::Granted request >Thread-48::DEBUG::2014-01-22 16:51:13,788::task::811::TaskManager.Task::(resourceAcquired) Task=`52dbcf71-0da9-4d9e-8c3b-ec45d12f2b4b`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (shared) >Thread-48::DEBUG::2014-01-22 16:51:13,788::task::974::TaskManager.Task::(_decref) Task=`52dbcf71-0da9-4d9e-8c3b-ec45d12f2b4b`::ref 1 aborting False >Thread-48::WARNING::2014-01-22 16:51:13,790::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5 already exists >Thread-48::DEBUG::2014-01-22 16:51:13,791::fileVolume::528::Storage.Volume::(validateVolumePath) validate path for fafd2cd2-6294-465b-b626-de8019782459 >Thread-48::INFO::2014-01-22 16:51:13,792::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459.lease', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, 'path': '/var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'imgVolumesInfo': [{'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459.lease', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}]} >Thread-48::DEBUG::2014-01-22 16:51:13,792::task::1168::TaskManager.Task::(prepare) Task=`52dbcf71-0da9-4d9e-8c3b-ec45d12f2b4b`::finished: {'info': {'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459.lease', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, 'path': '/var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'imgVolumesInfo': [{'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459.lease', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}]} >Thread-48::DEBUG::2014-01-22 16:51:13,792::task::579::TaskManager.Task::(_updateState) Task=`52dbcf71-0da9-4d9e-8c3b-ec45d12f2b4b`::moving from state preparing -> state finished >Thread-48::DEBUG::2014-01-22 16:51:13,793::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>} >Thread-48::DEBUG::2014-01-22 16:51:13,793::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-48::DEBUG::2014-01-22 16:51:13,793::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >Thread-48::DEBUG::2014-01-22 16:51:13,793::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >Thread-48::DEBUG::2014-01-22 16:51:13,793::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >Thread-48::DEBUG::2014-01-22 16:51:13,793::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >Thread-48::DEBUG::2014-01-22 16:51:13,794::task::974::TaskManager.Task::(_decref) Task=`52dbcf71-0da9-4d9e-8c3b-ec45d12f2b4b`::ref 0 aborting False >Thread-48::INFO::2014-01-22 16:51:13,794::clientIF::353::vds::(prepareVolumePath) prepared volume path: /var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459 >Thread-48::DEBUG::2014-01-22 16:51:13,829::utils::489::root::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_hostedengine' (cwd None) >Thread-48::DEBUG::2014-01-22 16:51:13,869::utils::509::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-48::INFO::2014-01-22 16:51:13,869::hooks::88::root::(_runHooksDir) >Thread-48::DEBUG::2014-01-22 16:51:13,869::vm::3031::vm.Vm::(_run) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::<?xml version="1.0" encoding="utf-8"?><domain type="kvm"> > <name>HostedEngine</name> > <uuid>bfc3da60-a697-4890-aa90-6d293b6237dc</uuid> > <memory>4194304</memory> > <currentMemory>4194304</currentMemory> > <vcpu>2</vcpu> > <memtune> > <min_guarantee>0</min_guarantee> > </memtune> > <devices> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/bfc3da60-a697-4890-aa90-6d293b6237dc.com.redhat.rhevm.vdsm"/> > </channel> > <channel type="unix"> > <target name="org.qemu.guest_agent.0" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/bfc3da60-a697-4890-aa90-6d293b6237dc.org.qemu.guest_agent.0"/> > </channel> > <input bus="ps2" type="mouse"/> > <graphics autoport="yes" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" type="vnc"/> > <controller model="virtio-scsi" type="scsi"/> > <interface type="bridge"> > <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/> > <mac address="00:16:3e:4e:70:25"/> > <model type="virtio"/> > <source bridge="rhevm"/> > <filterref filter="vdsm-no-mac-spoofing"/> > <link state="up"/> > <boot order="1"/> > </interface> > <disk device="cdrom" snapshot="no" type="file"> > <address bus="1" controller="0" target="0" type="drive" unit="0"/> > <source file="" startupPolicy="optional"/> > <target bus="ide" dev="hdc"/> > <readonly/> > <serial/> > </disk> > <disk device="disk" snapshot="no" type="file"> > <address bus="0x00" domain="0x0000" function="0x0" slot="0x06" type="pci"/> > <source file="/var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459"/> > <target bus="virtio" dev="vda"/> > <serial>e836fc1b-612b-447a-a52a-60db85dd9699</serial> > <driver cache="none" error_policy="stop" io="threads" name="qemu" type="raw"/> > </disk> > <console type="pty"> > <target port="0" type="virtio"/> > </console> > <memballoon model="none"/> > <lease> > <key>fafd2cd2-6294-465b-b626-de8019782459</key> > <lockspace>3900e952-713e-4ea9-b32c-097cd4994bc5</lockspace> > <target offset="0" path="/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459.lease"/> > </lease> > </devices> > <os> > <type arch="x86_64" machine="rhel6.5.0">hvm</type> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6Server-6.5.0.1.el6</entry> > <entry name="serial">4C4C4544-0034-5310-8052-B3C04F4A354A_d4:ae:52:c7:1a:39</entry> > <entry name="uuid">bfc3da60-a697-4890-aa90-6d293b6237dc</entry> > </system> > </sysinfo> > <clock adjustment="0" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > </cpu> ><on_poweroff>destroy</on_poweroff><on_reboot>destroy</on_reboot><on_crash>destroy</on_crash></domain> >Thread-50::DEBUG::2014-01-22 16:51:13,918::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('bfc3da60-a697-4890-aa90-6d293b6237dc', '6352WREb', '10800', 'disconnect', {}) {} >Thread-50::ERROR::2014-01-22 16:51:13,919::BindingXMLRPC::1003::vds::(wrapper) unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/BindingXMLRPC.py", line 989, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/BindingXMLRPC.py", line 240, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 592, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/vm.py", line 4328, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ >AttributeError: 'NoneType' object has no attribute 'XMLDesc' >libvirtEventLoop::DEBUG::2014-01-22 16:51:14,573::vm::4943::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::event Started detail 0 opaque None >Thread-48::DEBUG::2014-01-22 16:51:14,617::sampling::285::vm.Vm::(start) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Start statistics collection >Thread-51::DEBUG::2014-01-22 16:51:14,617::sampling::314::vm.Vm::(run) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Stats thread started >Thread-51::DEBUG::2014-01-22 16:51:14,618::task::579::TaskManager.Task::(_updateState) Task=`77f20500-37b9-4e67-b8d7-4cfac702567a`::moving from state init -> state preparing >Thread-51::INFO::2014-01-22 16:51:14,618::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', volUUID='fafd2cd2-6294-465b-b626-de8019782459', options=None) >Thread-51::INFO::2014-01-22 16:51:14,620::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:51:14,620::task::1168::TaskManager.Task::(prepare) Task=`77f20500-37b9-4e67-b8d7-4cfac702567a`::finished: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:51:14,620::task::579::TaskManager.Task::(_updateState) Task=`77f20500-37b9-4e67-b8d7-4cfac702567a`::moving from state preparing -> state finished >Thread-51::DEBUG::2014-01-22 16:51:14,620::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-51::DEBUG::2014-01-22 16:51:14,620::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-51::DEBUG::2014-01-22 16:51:14,620::task::974::TaskManager.Task::(_decref) Task=`77f20500-37b9-4e67-b8d7-4cfac702567a`::ref 0 aborting False >Thread-48::DEBUG::2014-01-22 16:51:14,627::vmChannels::194::vds::(register) Add fileno 7 to listener's channels. >Thread-48::WARNING::2014-01-22 16:51:14,628::vm::3536::vm.Vm::(_readPauseCode) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::_readPauseCode unsupported by libvirt vm >Thread-48::DEBUG::2014-01-22 16:51:14,673::vm::2136::vm.Vm::(_startUnderlyingVm) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::_ongoingCreations released >Thread-52::DEBUG::2014-01-22 16:51:15,070::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmSetTicket with ('bfc3da60-a697-4890-aa90-6d293b6237dc', '6352WREb', '10800', 'disconnect', {}) {} >Thread-52::DEBUG::2014-01-22 16:51:15,107::BindingXMLRPC::991::vds::(wrapper) return vmSetTicket with {'status': {'message': 'Done', 'code': 0}} >Thread-53::DEBUG::2014-01-22 16:51:15,167::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-53::DEBUG::2014-01-22 16:51:15,167::vm::595::vm.Vm::(_getCpuStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::CPU stats not available: 'NoneType' object is unsubscriptable >Thread-53::DEBUG::2014-01-22 16:51:15,168::vm::629::vm.Vm::(_getNetworkStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Network stats not available >Thread-53::DEBUG::2014-01-22 16:51:15,168::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-53::DEBUG::2014-01-22 16:51:15,168::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-53::DEBUG::2014-01-22 16:51:15,168::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-53::DEBUG::2014-01-22 16:51:15,168::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-53::DEBUG::2014-01-22 16:51:15,168::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'state': 'unknown', 'macAddr': '00:16:3e:4e:70:25', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.00', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.55', 'elapsedTime': '1', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >VM Channels Listener::DEBUG::2014-01-22 16:51:15,510::vmChannels::102::vds::(_do_add_channels) fileno 7 was added to unconnected channels. >VM Channels Listener::DEBUG::2014-01-22 16:51:15,510::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 7. >VM Channels Listener::DEBUG::2014-01-22 16:51:15,510::guestIF::147::vm.Vm::(_connect) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Attempting connection to /var/lib/libvirt/qemu/channels/bfc3da60-a697-4890-aa90-6d293b6237dc.com.redhat.rhevm.vdsm >VM Channels Listener::DEBUG::2014-01-22 16:51:15,512::guestIF::150::vm.Vm::(_connect) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Connected to /var/lib/libvirt/qemu/channels/bfc3da60-a697-4890-aa90-6d293b6237dc.com.redhat.rhevm.vdsm >VM Channels Listener::DEBUG::2014-01-22 16:51:15,513::vmChannels::144::vds::(_handle_unconnected) Connecting to fileno 7 succeeded. >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,538::vm::595::vm.Vm::(_getCpuStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::CPU stats not available: 'NoneType' object is unsubscriptable >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,538::vm::629::vm.Vm::(_getNetworkStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Network stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,539::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,539::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,539::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,539::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,540::vm::595::vm.Vm::(_getCpuStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::CPU stats not available: 'NoneType' object is unsubscriptable >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,540::vm::629::vm.Vm::(_getNetworkStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Network stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,540::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,541::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,541::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:16,541::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-37::DEBUG::2014-01-22 16:51:17,619::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:51:17,626::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000341938 s, 1.4 MB/s\n'; <rc> = 0 >Thread-54::DEBUG::2014-01-22 16:51:20,227::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-54::DEBUG::2014-01-22 16:51:20,227::vm::595::vm.Vm::(_getCpuStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::CPU stats not available: 'NoneType' object is unsubscriptable >Thread-54::DEBUG::2014-01-22 16:51:20,227::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-54::DEBUG::2014-01-22 16:51:20,227::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-54::DEBUG::2014-01-22 16:51:20,227::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-54::DEBUG::2014-01-22 16:51:20,227::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-54::DEBUG::2014-01-22 16:51:20,228::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.00', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.52', 'elapsedTime': '6', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-55::DEBUG::2014-01-22 16:51:25,281::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-55::DEBUG::2014-01-22 16:51:25,281::vm::595::vm.Vm::(_getCpuStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::CPU stats not available: 'NoneType' object is unsubscriptable >Thread-55::DEBUG::2014-01-22 16:51:25,281::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-55::DEBUG::2014-01-22 16:51:25,281::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-55::DEBUG::2014-01-22 16:51:25,281::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-55::DEBUG::2014-01-22 16:51:25,281::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-55::DEBUG::2014-01-22 16:51:25,282::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.00', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.57', 'elapsedTime': '11', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,549::vm::595::vm.Vm::(_getCpuStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::CPU stats not available: 'NoneType' object is unsubscriptable >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,550::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,550::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,550::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,550::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,551::vm::595::vm.Vm::(_getCpuStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::CPU stats not available: 'NoneType' object is unsubscriptable >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,551::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,551::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,552::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:26,552::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-37::DEBUG::2014-01-22 16:51:27,639::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:51:27,644::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000274103 s, 1.8 MB/s\n'; <rc> = 0 >Thread-56::DEBUG::2014-01-22 16:51:30,336::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-56::DEBUG::2014-01-22 16:51:30,336::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-56::DEBUG::2014-01-22 16:51:30,336::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-56::DEBUG::2014-01-22 16:51:30,336::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-56::DEBUG::2014-01-22 16:51:30,336::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-56::DEBUG::2014-01-22 16:51:30,337::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '42.36', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.63', 'elapsedTime': '16', 'vmType': 'kvm', 'cpuSys': '31.76', 'appsList': [], 'guestIPs': ''}]} >libvirtEventLoop::DEBUG::2014-01-22 16:51:34,126::clientIF::586::vm.Vm::(dispatchLibvirtEvents) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::graphics event phase 0 localAddr {'node': '0.0.0.0', 'service': '5900', 'family': 0} remoteAddr {'node': '127.0.0.1', 'service': '51258', 'family': 0}authScheme vnc subject [] >Thread-57::DEBUG::2014-01-22 16:51:35,402::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-57::DEBUG::2014-01-22 16:51:35,402::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-57::DEBUG::2014-01-22 16:51:35,402::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-57::DEBUG::2014-01-22 16:51:35,402::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-57::DEBUG::2014-01-22 16:51:35,402::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-57::DEBUG::2014-01-22 16:51:35,403::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '42.36', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.69', 'elapsedTime': '21', 'vmType': 'kvm', 'cpuSys': '31.76', 'appsList': [], 'guestIPs': ''}]} >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:36,561::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:36,561::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:36,561::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:36,561::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:36,561::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:36,561::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:36,562::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:36,562::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-37::DEBUG::2014-01-22 16:51:37,660::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:51:37,668::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000324533 s, 1.5 MB/s\n'; <rc> = 0 >Thread-58::DEBUG::2014-01-22 16:51:40,460::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-58::DEBUG::2014-01-22 16:51:40,461::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-58::DEBUG::2014-01-22 16:51:40,461::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-58::DEBUG::2014-01-22 16:51:40,462::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-58::DEBUG::2014-01-22 16:51:40,462::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-58::DEBUG::2014-01-22 16:51:40,462::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '42.36', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.74', 'elapsedTime': '26', 'vmType': 'kvm', 'cpuSys': '31.76', 'appsList': [], 'guestIPs': ''}]} >Thread-59::DEBUG::2014-01-22 16:51:45,524::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-59::DEBUG::2014-01-22 16:51:45,525::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-59::DEBUG::2014-01-22 16:51:45,525::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-59::DEBUG::2014-01-22 16:51:45,525::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-59::DEBUG::2014-01-22 16:51:45,525::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-59::DEBUG::2014-01-22 16:51:45,526::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '30.73', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.80', 'elapsedTime': '31', 'vmType': 'kvm', 'cpuSys': '22.17', 'appsList': [], 'guestIPs': ''}]} >VM Channels Listener::DEBUG::2014-01-22 16:51:45,546::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:46,572::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:46,573::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:46,573::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:46,573::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:46,574::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:46,574::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:46,574::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:46,574::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >libvirtEventLoop::DEBUG::2014-01-22 16:51:47,583::clientIF::586::vm.Vm::(dispatchLibvirtEvents) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::graphics event phase 1 localAddr {'node': '0.0.0.0', 'service': '5900', 'family': 0} remoteAddr {'node': '127.0.0.1', 'service': '51258', 'family': 0}authScheme vnc subject [] >Thread-37::DEBUG::2014-01-22 16:51:47,684::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:51:47,691::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000403544 s, 1.2 MB/s\n'; <rc> = 0 >Thread-60::DEBUG::2014-01-22 16:51:50,585::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-60::DEBUG::2014-01-22 16:51:50,586::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-60::DEBUG::2014-01-22 16:51:50,586::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-60::DEBUG::2014-01-22 16:51:50,586::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-60::DEBUG::2014-01-22 16:51:50,586::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-60::DEBUG::2014-01-22 16:51:50,587::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '30.73', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.85', 'elapsedTime': '36', 'vmType': 'kvm', 'cpuSys': '22.17', 'appsList': [], 'guestIPs': ''}]} >Thread-61::DEBUG::2014-01-22 16:51:55,646::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-61::DEBUG::2014-01-22 16:51:55,647::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-61::DEBUG::2014-01-22 16:51:55,647::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-61::DEBUG::2014-01-22 16:51:55,647::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-61::DEBUG::2014-01-22 16:51:55,647::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-61::DEBUG::2014-01-22 16:51:55,648::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '30.73', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.91', 'elapsedTime': '41', 'vmType': 'kvm', 'cpuSys': '22.17', 'appsList': [], 'guestIPs': ''}]} >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:56,583::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:56,584::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:56,584::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:56,584::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:56,585::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:56,585::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:56,586::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:51:56,586::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-37::DEBUG::2014-01-22 16:51:57,706::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:51:57,712::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000301724 s, 1.6 MB/s\n'; <rc> = 0 >Thread-62::DEBUG::2014-01-22 16:52:00,707::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-62::DEBUG::2014-01-22 16:52:00,708::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-62::DEBUG::2014-01-22 16:52:00,708::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-62::DEBUG::2014-01-22 16:52:00,708::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-62::DEBUG::2014-01-22 16:52:00,708::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-62::DEBUG::2014-01-22 16:52:00,708::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.22', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.97', 'elapsedTime': '46', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-63::DEBUG::2014-01-22 16:52:05,767::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-63::DEBUG::2014-01-22 16:52:05,768::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-63::DEBUG::2014-01-22 16:52:05,768::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-63::DEBUG::2014-01-22 16:52:05,768::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-63::DEBUG::2014-01-22 16:52:05,768::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-63::DEBUG::2014-01-22 16:52:05,768::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.22', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '1.02', 'elapsedTime': '52', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:52:06,595::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:52:06,596::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:52:06,596::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:52:06,596::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:52:06,597::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:52:06,597::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:52:06,597::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >GuestMonitor-HostedEngine::DEBUG::2014-01-22 16:52:06,597::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-37::DEBUG::2014-01-22 16:52:07,726::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:52:07,732::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000292302 s, 1.6 MB/s\n'; <rc> = 0 >Thread-64::DEBUG::2014-01-22 16:52:10,827::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-64::DEBUG::2014-01-22 16:52:10,827::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc stats not available >Thread-64::DEBUG::2014-01-22 16:52:10,828::vm::649::vm.Vm::(_getDiskStats) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda stats not available >Thread-64::DEBUG::2014-01-22 16:52:10,828::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk hdc latency not available >Thread-64::DEBUG::2014-01-22 16:52:10,828::vm::688::vm.Vm::(_getDiskLatency) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Disk vda latency not available >Thread-64::DEBUG::2014-01-22 16:52:10,828::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.22', 'disks': {u'vda': {'truesize': '24576', 'apparentsize': '26843545600', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, u'hdc': {'truesize': '0', 'apparentsize': '0'}}, 'monitorResponse': '0', 'statsAge': '0.08', 'elapsedTime': '57', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-51::DEBUG::2014-01-22 16:52:14,751::task::579::TaskManager.Task::(_updateState) Task=`9ba8a9e9-6c8f-4b17-b25f-9d739f12093d`::moving from state init -> state preparing >Thread-51::INFO::2014-01-22 16:52:14,752::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', volUUID='fafd2cd2-6294-465b-b626-de8019782459', options=None) >Thread-51::INFO::2014-01-22 16:52:14,757::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:52:14,757::task::1168::TaskManager.Task::(prepare) Task=`9ba8a9e9-6c8f-4b17-b25f-9d739f12093d`::finished: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:52:14,757::task::579::TaskManager.Task::(_updateState) Task=`9ba8a9e9-6c8f-4b17-b25f-9d739f12093d`::moving from state preparing -> state finished >Thread-51::DEBUG::2014-01-22 16:52:14,757::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-51::DEBUG::2014-01-22 16:52:14,757::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-51::DEBUG::2014-01-22 16:52:14,758::task::974::TaskManager.Task::(_decref) Task=`9ba8a9e9-6c8f-4b17-b25f-9d739f12093d`::ref 0 aborting False >VM Channels Listener::DEBUG::2014-01-22 16:52:15,579::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-65::DEBUG::2014-01-22 16:52:15,887::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-65::DEBUG::2014-01-22 16:52:15,888::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.18', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.14', 'elapsedTime': '62', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:52:17,747::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:52:17,754::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000347964 s, 1.4 MB/s\n'; <rc> = 0 >Thread-66::DEBUG::2014-01-22 16:52:20,947::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-66::DEBUG::2014-01-22 16:52:20,948::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.18', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.16', 'elapsedTime': '67', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-67::DEBUG::2014-01-22 16:52:26,008::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-67::DEBUG::2014-01-22 16:52:26,009::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.18', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.22', 'elapsedTime': '72', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:52:27,770::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:52:27,777::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000395838 s, 1.2 MB/s\n'; <rc> = 0 >Thread-68::DEBUG::2014-01-22 16:52:31,069::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-68::DEBUG::2014-01-22 16:52:31,070::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.05', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.27', 'elapsedTime': '77', 'vmType': 'kvm', 'cpuSys': '0.20', 'appsList': [], 'guestIPs': ''}]} >Thread-69::DEBUG::2014-01-22 16:52:36,130::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-69::DEBUG::2014-01-22 16:52:36,131::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.05', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.33', 'elapsedTime': '82', 'vmType': 'kvm', 'cpuSys': '0.20', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:52:37,792::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:52:37,799::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000366809 s, 1.3 MB/s\n'; <rc> = 0 >Thread-70::DEBUG::2014-01-22 16:52:41,191::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-70::DEBUG::2014-01-22 16:52:41,192::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.05', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.39', 'elapsedTime': '87', 'vmType': 'kvm', 'cpuSys': '0.20', 'appsList': [], 'guestIPs': ''}]} >VM Channels Listener::DEBUG::2014-01-22 16:52:45,612::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-71::DEBUG::2014-01-22 16:52:46,251::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-71::DEBUG::2014-01-22 16:52:46,252::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.18', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.45', 'elapsedTime': '92', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:52:47,813::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:52:47,820::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000338498 s, 1.4 MB/s\n'; <rc> = 0 >Thread-72::DEBUG::2014-01-22 16:52:51,311::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-72::DEBUG::2014-01-22 16:52:51,312::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.18', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.50', 'elapsedTime': '97', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-73::DEBUG::2014-01-22 16:52:56,369::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-73::DEBUG::2014-01-22 16:52:56,369::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '0.18', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.55', 'elapsedTime': '102', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:52:57,833::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:52:57,837::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000303006 s, 1.6 MB/s\n'; <rc> = 0 >Thread-74::DEBUG::2014-01-22 16:53:01,428::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-74::DEBUG::2014-01-22 16:53:01,429::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '19.36', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.60', 'elapsedTime': '107', 'vmType': 'kvm', 'cpuSys': '14.52', 'appsList': [], 'guestIPs': ''}]} >Thread-75::DEBUG::2014-01-22 16:53:06,485::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-75::DEBUG::2014-01-22 16:53:06,485::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '19.36', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.66', 'elapsedTime': '112', 'vmType': 'kvm', 'cpuSys': '14.52', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:53:07,847::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:53:07,852::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000272822 s, 1.8 MB/s\n'; <rc> = 0 >Thread-76::DEBUG::2014-01-22 16:53:11,540::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-76::DEBUG::2014-01-22 16:53:11,540::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.1', 'rxRate': '3.1', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '19.36', 'disks': {u'vda': {'readLatency': '594679', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.51', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.71', 'elapsedTime': '117', 'vmType': 'kvm', 'cpuSys': '14.52', 'appsList': [], 'guestIPs': ''}]} >Thread-51::DEBUG::2014-01-22 16:53:14,830::task::579::TaskManager.Task::(_updateState) Task=`5c70d1a0-1f9b-4b61-9989-748e31f3d187`::moving from state init -> state preparing >Thread-51::INFO::2014-01-22 16:53:14,831::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', volUUID='fafd2cd2-6294-465b-b626-de8019782459', options=None) >Thread-51::INFO::2014-01-22 16:53:14,834::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:53:14,834::task::1168::TaskManager.Task::(prepare) Task=`5c70d1a0-1f9b-4b61-9989-748e31f3d187`::finished: {'truesize': '24576', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:53:14,834::task::579::TaskManager.Task::(_updateState) Task=`5c70d1a0-1f9b-4b61-9989-748e31f3d187`::moving from state preparing -> state finished >Thread-51::DEBUG::2014-01-22 16:53:14,834::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-51::DEBUG::2014-01-22 16:53:14,834::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-51::DEBUG::2014-01-22 16:53:14,835::task::974::TaskManager.Task::(_decref) Task=`5c70d1a0-1f9b-4b61-9989-748e31f3d187`::ref 0 aborting False >VM Channels Listener::DEBUG::2014-01-22 16:53:15,645::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-77::DEBUG::2014-01-22 16:53:16,596::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-77::DEBUG::2014-01-22 16:53:16,596::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.1', 'rxRate': '3.1', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '45.63', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.77', 'elapsedTime': '122', 'vmType': 'kvm', 'cpuSys': '32.77', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:53:17,866::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:53:17,873::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000287335 s, 1.7 MB/s\n'; <rc> = 0 >Thread-78::DEBUG::2014-01-22 16:53:21,651::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-78::DEBUG::2014-01-22 16:53:21,652::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '45.63', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.79', 'elapsedTime': '127', 'vmType': 'kvm', 'cpuSys': '32.77', 'appsList': [], 'guestIPs': ''}]} >Thread-79::DEBUG::2014-01-22 16:53:26,711::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-79::DEBUG::2014-01-22 16:53:26,711::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '45.63', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.85', 'elapsedTime': '132', 'vmType': 'kvm', 'cpuSys': '32.77', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:53:27,888::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:53:27,895::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000340137 s, 1.4 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-01-22 16:53:31,771::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-80::DEBUG::2014-01-22 16:53:31,772::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '24.19', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.90', 'elapsedTime': '138', 'vmType': 'kvm', 'cpuSys': '1.53', 'appsList': [], 'guestIPs': ''}]} >Thread-81::DEBUG::2014-01-22 16:53:36,825::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-81::DEBUG::2014-01-22 16:53:36,825::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '24.19', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.96', 'elapsedTime': '143', 'vmType': 'kvm', 'cpuSys': '1.53', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:53:37,906::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:53:37,911::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000305311 s, 1.6 MB/s\n'; <rc> = 0 >Thread-82::DEBUG::2014-01-22 16:53:41,883::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-82::DEBUG::2014-01-22 16:53:41,883::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.1', 'rxRate': '24.4', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '24.19', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.01', 'elapsedTime': '148', 'vmType': 'kvm', 'cpuSys': '1.53', 'appsList': [], 'guestIPs': ''}]} >VM Channels Listener::DEBUG::2014-01-22 16:53:45,679::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-83::DEBUG::2014-01-22 16:53:46,942::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-83::DEBUG::2014-01-22 16:53:46,943::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '35.82', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.06', 'elapsedTime': '153', 'vmType': 'kvm', 'cpuSys': '7.19', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:53:47,929::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:53:47,962::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.0148643 s, 32.3 kB/s\n'; <rc> = 0 >Thread-84::DEBUG::2014-01-22 16:53:51,999::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-84::DEBUG::2014-01-22 16:53:51,999::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '35.82', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.11', 'elapsedTime': '158', 'vmType': 'kvm', 'cpuSys': '7.19', 'appsList': [], 'guestIPs': ''}]} >Thread-85::DEBUG::2014-01-22 16:53:57,058::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-85::DEBUG::2014-01-22 16:53:57,058::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '1.4', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '35.82', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.17', 'elapsedTime': '163', 'vmType': 'kvm', 'cpuSys': '7.19', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:53:57,982::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:53:57,989::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000346139 s, 1.4 MB/s\n'; <rc> = 0 >Thread-86::DEBUG::2014-01-22 16:54:02,116::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-86::DEBUG::2014-01-22 16:54:02,116::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.3', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '13.63', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.22', 'elapsedTime': '168', 'vmType': 'kvm', 'cpuSys': '2.40', 'appsList': [], 'guestIPs': ''}]} >Thread-87::DEBUG::2014-01-22 16:54:07,173::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-87::DEBUG::2014-01-22 16:54:07,173::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '2.1', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '13.63', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.28', 'elapsedTime': '173', 'vmType': 'kvm', 'cpuSys': '2.40', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:54:08,002::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:54:08,007::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000241173 s, 2.0 MB/s\n'; <rc> = 0 >Thread-88::DEBUG::2014-01-22 16:54:12,228::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-88::DEBUG::2014-01-22 16:54:12,228::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.1', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '13.63', 'disks': {u'vda': {'readLatency': '755176', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '0', 'readRate': '8.52', 'truesize': '24576', 'writeRate': '0.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.33', 'elapsedTime': '178', 'vmType': 'kvm', 'cpuSys': '2.40', 'appsList': [], 'guestIPs': ''}]} >Thread-51::DEBUG::2014-01-22 16:54:14,903::task::579::TaskManager.Task::(_updateState) Task=`51870ae5-f94d-4e2d-bd58-5f3866ea600b`::moving from state init -> state preparing >Thread-51::INFO::2014-01-22 16:54:14,903::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', volUUID='fafd2cd2-6294-465b-b626-de8019782459', options=None) >Thread-51::INFO::2014-01-22 16:54:14,907::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '546603008', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:54:14,908::task::1168::TaskManager.Task::(prepare) Task=`51870ae5-f94d-4e2d-bd58-5f3866ea600b`::finished: {'truesize': '546603008', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:54:14,908::task::579::TaskManager.Task::(_updateState) Task=`51870ae5-f94d-4e2d-bd58-5f3866ea600b`::moving from state preparing -> state finished >Thread-51::DEBUG::2014-01-22 16:54:14,908::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-51::DEBUG::2014-01-22 16:54:14,908::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-51::DEBUG::2014-01-22 16:54:14,908::task::974::TaskManager.Task::(_decref) Task=`51870ae5-f94d-4e2d-bd58-5f3866ea600b`::ref 0 aborting False >VM Channels Listener::DEBUG::2014-01-22 16:54:15,712::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-89::DEBUG::2014-01-22 16:54:17,287::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-89::DEBUG::2014-01-22 16:54:17,288::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '2.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '80.48', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.36', 'elapsedTime': '183', 'vmType': 'kvm', 'cpuSys': '0.60', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:54:18,015::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:54:18,020::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000283915 s, 1.7 MB/s\n'; <rc> = 0 >Thread-90::DEBUG::2014-01-22 16:54:22,345::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-90::DEBUG::2014-01-22 16:54:22,345::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.1', 'rxRate': '5.7', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '80.48', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.42', 'elapsedTime': '188', 'vmType': 'kvm', 'cpuSys': '0.60', 'appsList': [], 'guestIPs': ''}]} >Thread-91::DEBUG::2014-01-22 16:54:27,403::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-91::DEBUG::2014-01-22 16:54:27,404::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.5', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '80.48', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.47', 'elapsedTime': '193', 'vmType': 'kvm', 'cpuSys': '0.60', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:54:28,031::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:54:28,036::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000359662 s, 1.3 MB/s\n'; <rc> = 0 >Thread-92::DEBUG::2014-01-22 16:54:32,460::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-92::DEBUG::2014-01-22 16:54:32,460::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '2.4', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '80.43', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.52', 'elapsedTime': '198', 'vmType': 'kvm', 'cpuSys': '5.99', 'appsList': [], 'guestIPs': ''}]} >Thread-93::DEBUG::2014-01-22 16:54:37,517::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-93::DEBUG::2014-01-22 16:54:37,517::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '3.2', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '80.43', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.58', 'elapsedTime': '203', 'vmType': 'kvm', 'cpuSys': '5.99', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:54:38,045::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:54:38,049::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000279166 s, 1.7 MB/s\n'; <rc> = 0 >Thread-94::DEBUG::2014-01-22 16:54:42,575::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-94::DEBUG::2014-01-22 16:54:42,576::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '3.5', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '80.43', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.63', 'elapsedTime': '208', 'vmType': 'kvm', 'cpuSys': '5.99', 'appsList': [], 'guestIPs': ''}]} >VM Channels Listener::DEBUG::2014-01-22 16:54:45,745::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-95::DEBUG::2014-01-22 16:54:47,633::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-95::DEBUG::2014-01-22 16:54:47,633::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '5.4', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '75.88', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.68', 'elapsedTime': '213', 'vmType': 'kvm', 'cpuSys': '1.93', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:54:48,057::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:54:48,061::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.00028777 s, 1.7 MB/s\n'; <rc> = 0 >Thread-96::DEBUG::2014-01-22 16:54:52,689::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-96::DEBUG::2014-01-22 16:54:52,690::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '3.1', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '75.88', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.74', 'elapsedTime': '218', 'vmType': 'kvm', 'cpuSys': '1.93', 'appsList': [], 'guestIPs': ''}]} >Thread-97::DEBUG::2014-01-22 16:54:57,749::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-97::DEBUG::2014-01-22 16:54:57,750::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '2.6', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '75.88', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.79', 'elapsedTime': '224', 'vmType': 'kvm', 'cpuSys': '1.93', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:54:58,076::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:54:58,088::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000622673 s, 771 kB/s\n'; <rc> = 0 >Thread-98::DEBUG::2014-01-22 16:55:02,811::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-98::DEBUG::2014-01-22 16:55:02,811::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.5', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '70.33', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.85', 'elapsedTime': '229', 'vmType': 'kvm', 'cpuSys': '5.13', 'appsList': [], 'guestIPs': ''}]} >Thread-99::DEBUG::2014-01-22 16:55:07,865::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-99::DEBUG::2014-01-22 16:55:07,865::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '3.1', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '70.33', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.90', 'elapsedTime': '234', 'vmType': 'kvm', 'cpuSys': '5.13', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:55:08,095::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:55:08,100::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000294428 s, 1.6 MB/s\n'; <rc> = 0 >Thread-100::DEBUG::2014-01-22 16:55:12,920::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-100::DEBUG::2014-01-22 16:55:12,920::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '5.2', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '70.33', 'disks': {u'vda': {'readLatency': '384866', 'apparentsize': '26843545600', 'writeLatency': '214220891', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '68106', 'readRate': '264638.02', 'truesize': '546603008', 'writeRate': '9180152.42'}, u'hdc': {'readLatency': '2885', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '7.79', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.95', 'elapsedTime': '239', 'vmType': 'kvm', 'cpuSys': '5.13', 'appsList': [], 'guestIPs': ''}]} >Thread-51::DEBUG::2014-01-22 16:55:14,969::task::579::TaskManager.Task::(_updateState) Task=`28a650a9-0bac-490a-8c13-6eb37aab11bb`::moving from state init -> state preparing >Thread-51::INFO::2014-01-22 16:55:14,969::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', volUUID='fafd2cd2-6294-465b-b626-de8019782459', options=None) >Thread-51::INFO::2014-01-22 16:55:14,971::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '1297121280', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:55:14,971::task::1168::TaskManager.Task::(prepare) Task=`28a650a9-0bac-490a-8c13-6eb37aab11bb`::finished: {'truesize': '1297121280', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:55:14,971::task::579::TaskManager.Task::(_updateState) Task=`28a650a9-0bac-490a-8c13-6eb37aab11bb`::moving from state preparing -> state finished >Thread-51::DEBUG::2014-01-22 16:55:14,971::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-51::DEBUG::2014-01-22 16:55:14,972::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-51::DEBUG::2014-01-22 16:55:14,972::task::974::TaskManager.Task::(_decref) Task=`28a650a9-0bac-490a-8c13-6eb37aab11bb`::ref 0 aborting False >VM Channels Listener::DEBUG::2014-01-22 16:55:15,778::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-101::DEBUG::2014-01-22 16:55:17,976::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-101::DEBUG::2014-01-22 16:55:17,976::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '87.35', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.99', 'elapsedTime': '244', 'vmType': 'kvm', 'cpuSys': '2.33', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:55:18,106::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:55:18,110::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000272723 s, 1.8 MB/s\n'; <rc> = 0 >Thread-102::DEBUG::2014-01-22 16:55:23,028::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-102::DEBUG::2014-01-22 16:55:23,028::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '87.35', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.04', 'elapsedTime': '249', 'vmType': 'kvm', 'cpuSys': '2.33', 'appsList': [], 'guestIPs': ''}]} >Thread-103::DEBUG::2014-01-22 16:55:28,083::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-103::DEBUG::2014-01-22 16:55:28,083::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '87.35', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.09', 'elapsedTime': '254', 'vmType': 'kvm', 'cpuSys': '2.33', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:55:28,119::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:55:28,125::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000280312 s, 1.7 MB/s\n'; <rc> = 0 >Thread-104::DEBUG::2014-01-22 16:55:33,136::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-104::DEBUG::2014-01-22 16:55:33,136::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '2.7', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '98.55', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.14', 'elapsedTime': '259', 'vmType': 'kvm', 'cpuSys': '1.06', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:55:38,128::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-37::DEBUG::2014-01-22 16:55:38,135::fileSD::137::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5 >Thread-37::DEBUG::2014-01-22 16:55:38,136::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-37::DEBUG::2014-01-22 16:55:38,144::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=hosted_storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=ON', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=hosted_datacenter', 'POOL_DOMAINS=3900e952-713e-4ea9-b32c-097cd4994bc5:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=1', 'POOL_UUID=e1263145-a6a9-4993-9d8a-cd92a3e5c1fb', 'REMOTE_PATH=10.34.63.199:/jbelka/hosted', 'ROLE=Master', 'SDUUID=3900e952-713e-4ea9-b32c-097cd4994bc5', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=e8504e783bbf704c2dbec50f6a1b369b8550bd07'] >Thread-37::DEBUG::2014-01-22 16:55:38,146::fileSD::558::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-37::INFO::2014-01-22 16:55:38,146::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_imageNS already registered >Thread-37::INFO::2014-01-22 16:55:38,146::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3900e952-713e-4ea9-b32c-097cd4994bc5_volumeNS already registered >Thread-37::DEBUG::2014-01-22 16:55:38,160::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:55:38,167::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000340759 s, 1.4 MB/s\n'; <rc> = 0 >Thread-106::DEBUG::2014-01-22 16:55:38,195::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-106::DEBUG::2014-01-22 16:55:38,196::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '2.7', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '98.55', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.19', 'elapsedTime': '264', 'vmType': 'kvm', 'cpuSys': '1.06', 'appsList': [], 'guestIPs': ''}]} >Thread-107::DEBUG::2014-01-22 16:55:43,256::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-107::DEBUG::2014-01-22 16:55:43,256::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '98.55', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.25', 'elapsedTime': '269', 'vmType': 'kvm', 'cpuSys': '1.06', 'appsList': [], 'guestIPs': ''}]} >VM Channels Listener::DEBUG::2014-01-22 16:55:45,809::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-37::DEBUG::2014-01-22 16:55:48,181::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:55:48,186::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000255223 s, 1.9 MB/s\n'; <rc> = 0 >Thread-108::DEBUG::2014-01-22 16:55:48,314::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-108::DEBUG::2014-01-22 16:55:48,315::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '103.54', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.30', 'elapsedTime': '274', 'vmType': 'kvm', 'cpuSys': '4.66', 'appsList': [], 'guestIPs': ''}]} >Thread-109::DEBUG::2014-01-22 16:55:53,371::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-109::DEBUG::2014-01-22 16:55:53,371::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '103.54', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.36', 'elapsedTime': '279', 'vmType': 'kvm', 'cpuSys': '4.66', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:55:58,193::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:55:58,198::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000267641 s, 1.8 MB/s\n'; <rc> = 0 >Thread-110::DEBUG::2014-01-22 16:55:58,427::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-110::DEBUG::2014-01-22 16:55:58,427::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '103.54', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.41', 'elapsedTime': '284', 'vmType': 'kvm', 'cpuSys': '4.66', 'appsList': [], 'guestIPs': ''}]} >Thread-111::DEBUG::2014-01-22 16:56:03,482::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-111::DEBUG::2014-01-22 16:56:03,483::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '103.04', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.46', 'elapsedTime': '289', 'vmType': 'kvm', 'cpuSys': '1.93', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:56:08,209::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:56:08,216::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000447279 s, 1.1 MB/s\n'; <rc> = 0 >Thread-112::DEBUG::2014-01-22 16:56:08,540::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-112::DEBUG::2014-01-22 16:56:08,541::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '103.04', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.52', 'elapsedTime': '294', 'vmType': 'kvm', 'cpuSys': '1.93', 'appsList': [], 'guestIPs': ''}]} >Thread-113::DEBUG::2014-01-22 16:56:13,602::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-113::DEBUG::2014-01-22 16:56:13,603::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '103.04', 'disks': {u'vda': {'readLatency': '37523527', 'apparentsize': '26843545600', 'writeLatency': '47746236', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '60679', 'readRate': '562.61', 'truesize': '1297121280', 'writeRate': '13725300.44'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.58', 'elapsedTime': '299', 'vmType': 'kvm', 'cpuSys': '1.93', 'appsList': [], 'guestIPs': ''}]} >Thread-51::DEBUG::2014-01-22 16:56:15,028::task::579::TaskManager.Task::(_updateState) Task=`81565cab-04b1-4abb-9e16-d62b4ada2355`::moving from state init -> state preparing >Thread-51::INFO::2014-01-22 16:56:15,028::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', volUUID='fafd2cd2-6294-465b-b626-de8019782459', options=None) >Thread-51::INFO::2014-01-22 16:56:15,031::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '1650319360', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:56:15,031::task::1168::TaskManager.Task::(prepare) Task=`81565cab-04b1-4abb-9e16-d62b4ada2355`::finished: {'truesize': '1650319360', 'apparentsize': '26843545600'} >Thread-51::DEBUG::2014-01-22 16:56:15,032::task::579::TaskManager.Task::(_updateState) Task=`81565cab-04b1-4abb-9e16-d62b4ada2355`::moving from state preparing -> state finished >Thread-51::DEBUG::2014-01-22 16:56:15,032::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-51::DEBUG::2014-01-22 16:56:15,032::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-51::DEBUG::2014-01-22 16:56:15,032::task::974::TaskManager.Task::(_decref) Task=`81565cab-04b1-4abb-9e16-d62b4ada2355`::ref 0 aborting False >VM Channels Listener::DEBUG::2014-01-22 16:56:15,842::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-37::DEBUG::2014-01-22 16:56:18,226::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:56:18,232::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000281013 s, 1.7 MB/s\n'; <rc> = 0 >Thread-114::DEBUG::2014-01-22 16:56:18,662::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-114::DEBUG::2014-01-22 16:56:18,663::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '47.11', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.61', 'elapsedTime': '304', 'vmType': 'kvm', 'cpuSys': '3.00', 'appsList': [], 'guestIPs': ''}]} >Thread-115::DEBUG::2014-01-22 16:56:23,720::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-115::DEBUG::2014-01-22 16:56:23,721::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '47.11', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.67', 'elapsedTime': '309', 'vmType': 'kvm', 'cpuSys': '3.00', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:56:28,244::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:56:28,251::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000365722 s, 1.3 MB/s\n'; <rc> = 0 >Thread-116::DEBUG::2014-01-22 16:56:28,786::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-116::DEBUG::2014-01-22 16:56:28,787::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '47.11', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.73', 'elapsedTime': '315', 'vmType': 'kvm', 'cpuSys': '3.00', 'appsList': [], 'guestIPs': ''}]} >Thread-117::DEBUG::2014-01-22 16:56:33,847::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-117::DEBUG::2014-01-22 16:56:33,847::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '57.23', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.78', 'elapsedTime': '320', 'vmType': 'kvm', 'cpuSys': '3.19', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:56:38,260::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:56:38,264::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000280811 s, 1.7 MB/s\n'; <rc> = 0 >Thread-118::DEBUG::2014-01-22 16:56:38,899::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-118::DEBUG::2014-01-22 16:56:38,899::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '2.8', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '57.23', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.83', 'elapsedTime': '325', 'vmType': 'kvm', 'cpuSys': '3.19', 'appsList': [], 'guestIPs': ''}]} >Thread-119::DEBUG::2014-01-22 16:56:43,951::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-119::DEBUG::2014-01-22 16:56:43,951::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '57.23', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.88', 'elapsedTime': '330', 'vmType': 'kvm', 'cpuSys': '3.19', 'appsList': [], 'guestIPs': ''}]} >VM Channels Listener::DEBUG::2014-01-22 16:56:45,873::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 7. >Thread-37::DEBUG::2014-01-22 16:56:48,274::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:56:48,281::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000337201 s, 1.4 MB/s\n'; <rc> = 0 >Thread-120::DEBUG::2014-01-22 16:56:49,009::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-120::DEBUG::2014-01-22 16:56:49,010::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '91.24', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.94', 'elapsedTime': '335', 'vmType': 'kvm', 'cpuSys': '1.27', 'appsList': [], 'guestIPs': ''}]} >Thread-121::DEBUG::2014-01-22 16:56:54,071::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-121::DEBUG::2014-01-22 16:56:54,072::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '1.1', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '91.24', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.99', 'elapsedTime': '340', 'vmType': 'kvm', 'cpuSys': '1.27', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:56:58,292::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:56:58,297::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000288145 s, 1.7 MB/s\n'; <rc> = 0 >Thread-122::DEBUG::2014-01-22 16:56:59,132::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-122::DEBUG::2014-01-22 16:56:59,133::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '1.2', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '91.24', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.05', 'elapsedTime': '345', 'vmType': 'kvm', 'cpuSys': '1.27', 'appsList': [], 'guestIPs': ''}]} >Thread-123::DEBUG::2014-01-22 16:57:04,193::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-123::DEBUG::2014-01-22 16:57:04,194::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.3', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '36.43', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.11', 'elapsedTime': '350', 'vmType': 'kvm', 'cpuSys': '2.33', 'appsList': [], 'guestIPs': ''}]} >Thread-37::DEBUG::2014-01-22 16:57:08,309::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:57:08,316::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000436558 s, 1.1 MB/s\n'; <rc> = 0 >Thread-124::DEBUG::2014-01-22 16:57:09,255::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-124::DEBUG::2014-01-22 16:57:09,256::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '6586', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0', 'hash': '-3422828476155001960', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4e:70:25', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'displayType': 'vnc', 'cpuUser': '36.43', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '72470690', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'flushLatency': '59761', 'readRate': '0.00', 'truesize': '1650319360', 'writeRate': '6866673.00'}, u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.16', 'elapsedTime': '355', 'vmType': 'kvm', 'cpuSys': '2.33', 'appsList': [], 'guestIPs': ''}]} >libvirtEventLoop::DEBUG::2014-01-22 16:57:12,649::vm::4943::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::event Shutdown detail 0 opaque None >VM Channels Listener::ERROR::2014-01-22 16:57:12,818::vmChannels::53::vds::(_handle_event) Received 00000019 on fileno 7 >VM Channels Listener::DEBUG::2014-01-22 16:57:12,825::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 7. >VM Channels Listener::DEBUG::2014-01-22 16:57:12,826::guestIF::147::vm.Vm::(_connect) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Attempting connection to /var/lib/libvirt/qemu/channels/bfc3da60-a697-4890-aa90-6d293b6237dc.com.redhat.rhevm.vdsm >VM Channels Listener::DEBUG::2014-01-22 16:57:12,828::guestIF::158::vm.Vm::(_connect) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Failed to connect to /var/lib/libvirt/qemu/channels/bfc3da60-a697-4890-aa90-6d293b6237dc.com.redhat.rhevm.vdsm with 111 >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,057::vm::4943::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::event Stopped detail 0 opaque None >libvirtEventLoop::INFO::2014-01-22 16:57:13,058::vm::2197::vm.Vm::(_onQemuDeath) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::underlying process disconnected >libvirtEventLoop::INFO::2014-01-22 16:57:13,058::vm::4423::vm.Vm::(releaseVm) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Release VM resources >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,064::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid 'bfc3da60-a697-4890-aa90-6d293b6237dc' >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,064::sampling::292::vm.Vm::(stop) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Stop statistics collection >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,064::vmChannels::205::vds::(unregister) Delete fileno 7 from listener. >Thread-51::DEBUG::2014-01-22 16:57:13,064::sampling::323::vm.Vm::(run) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Stats thread finished >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,065::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid 'bfc3da60-a697-4890-aa90-6d293b6237dc' >libvirtEventLoop::WARNING::2014-01-22 16:57:13,066::clientIF::365::vds::(teardownVolumePath) Drive is not a vdsm image: VOLWM_CHUNK_MB:1024 VOLWM_CHUNK_REPLICATE_MULT:2 VOLWM_FREE_PCT:50 _blockDev:False _checkIoTuneCategories:<bound method Drive._checkIoTuneCategories of <vm.Drive object at 0x2140790>> _customize:<bound method Drive._customize of <vm.Drive object at 0x2140790>> _deviceXML:<disk device="cdrom" snapshot="no" type="file"><address bus="1" controller="0" target="0" type="drive" unit="0"/><source file="" startupPolicy="optional"/><target bus="ide" dev="hdc"/><readonly/><serial></serial></disk> _makeName:<bound method Drive._makeName of <vm.Drive object at 0x2140790>> _setExtSharedState:<bound method Drive._setExtSharedState of <vm.Drive object at 0x2140790>> _validateIoTuneParams:<bound method Drive._validateIoTuneParams of <vm.Drive object at 0x2140790>> address:{u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'} alias:ide0-1-0 apparentsize:0 blockDev:False cache:none conf:{'status': 'Up', 'nicModel': 'rtl8139,pv', 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'emulatedMachine': 'rhel6.5.0', 'displayIp': '0', 'pid': '6586', 'devices': [{'device': 'scsi', 'alias': u'scsi0', 'model': 'virtio-scsi', 'type': 'controller', 'address': {u'slot': u'0x04', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}}, {'nicModel': 'pv', 'macAddr': '00:16:3e:4e:70:25', 'linkActive': True, 'network': 'rhevm', 'alias': u'net0', 'bootOrder': '1', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': 'c42e7bca-ecce-4890-8ad9-cf42acd2c9eb', 'address': {u'slot': u'0x03', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'device': 'bridge', 'type': 'interface', 'name': u'vnet0'}, {'index': '2', 'iface': 'ide', 'name': u'hdc', 'alias': u'ide0-1-0', 'specParams': {}, 'readonly': 'True', 'deviceId': 'ffb0a6bb-a46c-4f3b-a892-c3a05741aca3', 'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'volumeInfo': {'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459.lease', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}, 'index': '0', 'iface': 'virtio', 'apparentsize': '26843545600', 'alias': u'virtio-disk0', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'readonly': 'False', 'shared': 'exclusive', 'truesize': '24576', 'type': 'disk', 'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'reqsize': '0', 'format': 'raw', 'deviceId': 'e836fc1b-612b-447a-a52a-60db85dd9699', 'poolID': '00000000-0000-0000-0000-000000000000', 'device': 'disk', 'path': '/var/run/vdsm/storage/3900e952-713e-4ea9-b32c-097cd4994bc5/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'propagateErrors': 'off', 'optional': 'false', 'name': u'vda', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'specParams': {}, 'volumeChain': [{'domainID': '3900e952-713e-4ea9-b32c-097cd4994bc5', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459', 'volumeID': 'fafd2cd2-6294-465b-b626-de8019782459', 'leasePath': '/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/images/e836fc1b-612b-447a-a52a-60db85dd9699/fafd2cd2-6294-465b-b626-de8019782459.lease', 'imageID': 'e836fc1b-612b-447a-a52a-60db85dd9699'}]}, {'device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': '24b23aec-7638-4133-a866-da3935c7c5dc', 'alias': 'console0'}, {'device': 'memballoon', 'specParams': {'model': 'none'}, 'type': 'balloon', 'alias': u'balloon0'}, {'device': u'usb', 'alias': u'usb0', 'type': 'controller', 'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x2'}}, {'device': u'ide', 'alias': u'ide0', 'type': 'controller', 'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x1'}}, {'device': u'virtio-serial', 'alias': u'virtio-serial0', 'type': 'controller', 'address': {u'slot': u'0x05', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}}, {'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}, {'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}, {'device': '', 'alias': u'video0', 'type': u'video', 'address': {u'slot': u'0x02', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}}], 'smp': '2', 'vmType': 'kvm', 'displaySecurePort': '-1', 'memSize': '4096', 'displayPort': u'5900', 'cpuType': 'Westmere', 'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'clientIp': '127.0.0.1', 'vmName': 'HostedEngine', 'display': 'vnc', 'pauseCode': 'NOERR'} createXmlElem:<bound method Drive.createXmlElem of <vm.Drive object at 0x2140790>> device:cdrom deviceId:ffb0a6bb-a46c-4f3b-a892-c3a05741aca3 drv:raw extSharedState:none getLeasesXML:<bound method Drive.getLeasesXML of <vm.Drive object at 0x2140790>> getNextVolumeSize:<bound method Drive.getNextVolumeSize of <vm.Drive object at 0x2140790>> getXML:<bound method Drive.getXML of <vm.Drive object at 0x2140790>> hasVolumeLeases:False iface:ide index:2 isDiskReplicationInProgress:<bound method Drive.isDiskReplicationInProgress of <vm.Drive object at 0x2140790>> isVdsmImage:<bound method Drive.isVdsmImage of <vm.Drive object at 0x2140790>> log:<logUtils.SimpleLogAdapter instance at 0x7f7a50129368> name:hdc networkDev:False path: readonly:True reqsize:0 serial: shared:false specParams:{} transientDisk:False truesize:0 type:cdrom volExtensionChunk:1024 watermarkLimit:536870912 >Traceback (most recent call last): > File "/usr/share/vdsm/clientIF.py", line 359, in teardownVolumePath > res = self.irs.teardownImage(drive['domainID'], > File "/usr/share/vdsm/vm.py", line 1406, in __getitem__ > raise KeyError(key) >KeyError: 'domainID' >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,068::task::579::TaskManager.Task::(_updateState) Task=`b41a49a1-ba34-400d-9896-315a3b6b6148`::moving from state init -> state preparing >libvirtEventLoop::INFO::2014-01-22 16:57:13,069::logUtils::44::dispatcher::(wrapper) Run and protect: teardownImage(sdUUID='3900e952-713e-4ea9-b32c-097cd4994bc5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='e836fc1b-612b-447a-a52a-60db85dd9699', volUUID=None) >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,069::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`f385dd04-4714-40f4-b69b-89afd1eca1c7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3304' at 'teardownImage' >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,069::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' for lock type 'shared' >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,070::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free. Now locking as 'shared' (1 active user) >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,070::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.3900e952-713e-4ea9-b32c-097cd4994bc5`ReqID=`f385dd04-4714-40f4-b69b-89afd1eca1c7`::Granted request >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,070::task::811::TaskManager.Task::(resourceAcquired) Task=`b41a49a1-ba34-400d-9896-315a3b6b6148`::_resourcesAcquired: Storage.3900e952-713e-4ea9-b32c-097cd4994bc5 (shared) >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,071::task::974::TaskManager.Task::(_decref) Task=`b41a49a1-ba34-400d-9896-315a3b6b6148`::ref 1 aborting False >libvirtEventLoop::INFO::2014-01-22 16:57:13,071::logUtils::47::dispatcher::(wrapper) Run and protect: teardownImage, Return response: None >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,071::task::1168::TaskManager.Task::(prepare) Task=`b41a49a1-ba34-400d-9896-315a3b6b6148`::finished: None >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,071::task::579::TaskManager.Task::(_updateState) Task=`b41a49a1-ba34-400d-9896-315a3b6b6148`::moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,071::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5': < ResourceRef 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', isValid: 'True' obj: 'None'>} >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,072::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,072::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,072::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' (0 active users) >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,072::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,073::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.3900e952-713e-4ea9-b32c-097cd4994bc5', Clearing records. >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,073::task::974::TaskManager.Task::(_decref) Task=`b41a49a1-ba34-400d-9896-315a3b6b6148`::ref 0 aborting False >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,073::task::579::TaskManager.Task::(_updateState) Task=`e6f2c48c-6fbc-4cf4-9aa7-25dd6952281d`::moving from state init -> state preparing >libvirtEventLoop::INFO::2014-01-22 16:57:13,074::logUtils::44::dispatcher::(wrapper) Run and protect: inappropriateDevices(thiefId='bfc3da60-a697-4890-aa90-6d293b6237dc') >libvirtEventLoop::INFO::2014-01-22 16:57:13,086::logUtils::47::dispatcher::(wrapper) Run and protect: inappropriateDevices, Return response: None >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,086::task::1168::TaskManager.Task::(prepare) Task=`e6f2c48c-6fbc-4cf4-9aa7-25dd6952281d`::finished: None >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,086::task::579::TaskManager.Task::(_updateState) Task=`e6f2c48c-6fbc-4cf4-9aa7-25dd6952281d`::moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,087::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,087::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,087::task::974::TaskManager.Task::(_decref) Task=`e6f2c48c-6fbc-4cf4-9aa7-25dd6952281d`::ref 0 aborting False >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,093::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid 'bfc3da60-a697-4890-aa90-6d293b6237dc' >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,093::vm::2615::vm.Vm::(setDownStatus) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Changed state to Down: User shut down >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,093::sampling::292::vm.Vm::(stop) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Stop statistics collection >libvirtEventLoop::DEBUG::2014-01-22 16:57:13,098::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid 'bfc3da60-a697-4890-aa90-6d293b6237dc' >VM Channels Listener::DEBUG::2014-01-22 16:57:13,829::vmChannels::112::vds::(_do_del_channels) fileno 7 was removed from listener. >Thread-125::DEBUG::2014-01-22 16:57:14,328::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-125::DEBUG::2014-01-22 16:57:14,328::BindingXMLRPC::991::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Down', 'vmId': 'bfc3da60-a697-4890-aa90-6d293b6237dc', 'hash': '-3422828476155001960', 'exitMessage': 'User shut down', 'exitCode': 0}]} >Thread-126::DEBUG::2014-01-22 16:57:14,474::BindingXMLRPC::984::vds::(wrapper) client [127.0.0.1]::call vmDestroy with ('bfc3da60-a697-4890-aa90-6d293b6237dc',) {} >Thread-126::INFO::2014-01-22 16:57:14,480::API::324::vds::(destroy) vmContainerLock acquired by vm bfc3da60-a697-4890-aa90-6d293b6237dc >Thread-126::DEBUG::2014-01-22 16:57:14,480::vm::4477::vm.Vm::(destroy) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::destroy Called >Thread-126::DEBUG::2014-01-22 16:57:14,480::vm::4471::vm.Vm::(deleteVm) vmId=`bfc3da60-a697-4890-aa90-6d293b6237dc`::Total desktops after destroy of bfc3da60-a697-4890-aa90-6d293b6237dc is 0 >Thread-126::DEBUG::2014-01-22 16:57:14,480::BindingXMLRPC::991::vds::(wrapper) return vmDestroy with {'status': {'message': 'Machine destroyed', 'code': 0}} >Thread-37::DEBUG::2014-01-22 16:57:18,330::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:57:18,336::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000410133 s, 1.2 MB/s\n'; <rc> = 0 >Thread-37::DEBUG::2014-01-22 16:57:28,349::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:57:28,356::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000467432 s, 1.0 MB/s\n'; <rc> = 0 >Thread-37::DEBUG::2014-01-22 16:57:38,370::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:57:38,378::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000459919 s, 1.0 MB/s\n'; <rc> = 0 >Thread-37::DEBUG::2014-01-22 16:57:48,391::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:57:48,398::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000431297 s, 1.1 MB/s\n'; <rc> = 0 >Thread-37::DEBUG::2014-01-22 16:57:58,411::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:57:58,418::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.00043336 s, 1.1 MB/s\n'; <rc> = 0 >Thread-37::DEBUG::2014-01-22 16:58:08,431::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:58:08,438::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000483351 s, 993 kB/s\n'; <rc> = 0 >Thread-37::DEBUG::2014-01-22 16:58:18,452::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.199:_jbelka_hosted/3900e952-713e-4ea9-b32c-097cd4994bc5/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-37::DEBUG::2014-01-22 16:58:18,459::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n480 bytes (480 B) copied, 0.000451759 s, 1.1 MB/s\n'; <rc> = 0
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1056649
: 853936